Announcement: I have spent the past few years writing a book on NIH, and it is under contract with MIT Press. The manuscript is in the editing process right now, and there is a ~7,000-word section on mental health that we’re going to cut for the sake of word count.
I’m reprinting it here, as it shows why NIH can be overly influenced by groupthink.
Thanks to Dorothy Bishop at Oxford for her comments!
***
At NIH, there’s one institute solely focused on mental health: NIMH, the National Institute of Mental Health. How has that research institute done at addressing mental illness in the US?
Not very well.
That’s according to Tom Insel himself, a long-time director of NIMH from 2002 through 2015.
As Insel told Wired magazine in 2017:
“I spent 13 years at NIMH really pushing on the neuroscience and genetics of mental disorders, and when I look back on that I realize that while I think I succeeded at getting lots of really cool papers published by cool scientists at fairly large costs—I think $20 billion—I don’t think we moved the needle in reducing suicide, reducing hospitalizations, improving recovery for the tens of millions of people who have mental illness.”[1]
Why would this be the case? NIMH was arguably too obsessed with one particular type of study, to the exclusion of others. That is, under Insel, NIMH turned its focus to studies that involved genetics or brain imaging, while downplaying studies on other factors (behavioral, societal, programs and interventions, etc.).
As the New York Times put it, “Government agencies, like the National Institute on Drug Abuse and the National Institute of Mental Health, continue to double down, sinking enormous sums of taxpayer money into biological research aimed at someday finding a neural signature or ‘blood test’ for psychiatric diagnoses that could be, maybe, one day in the future, useful — all while people are in crisis now.”[2]
The Times quotes a book by Insel: “The scientific progress in our field was stunning, but while we studied the risk factors for suicide, the death rate had climbed 33 percent. While we identified the neuroanatomy of addiction, overdose deaths had increased by threefold. While we mapped the genes for schizophrenia, people with this disease were still chronically unemployed and dying 20 years early.”
As Josh Dubnau, a neurobiologist at Stony Brook University, has said, “Many of us have been crying foul for years.”[3] Or as Eric Turkheimer, one of the most prominent behavioral geneticists in the US, has said, “I find Insel’s late career revelation that neurogenomics may not be the answer to mental illness profoundly infuriating. How many dollars were wasted while behavioral models were ignored at NIMH? Careers? Patient lives?”[4]
As Brad Wyble of Pennsylvania State noted, “the problem is that we don’t know what was lost, or how good behavioral work could have complemented neuro & genetics.”[5] And as John Krakauer, professor of neurology and neuroscience at Johns Hopkins said, “the neglect of behavioral research is a source of great shame. This belief in genes and magic bullets is a cult.”[6]
In other words, the NIH spent some $20 billion on 13 years of research that didn’t make any actual progress against our nation’s enormous mental health problem. And the person in charge may be largely to blame here, by his own account!
Thus, we have a story of how one NIH Institute Director can create groupthink in an entire field. The lesson is that NIH should have tried to create structural alternatives so that one person’s view (however enlightened and wise it might have seemed at the time) can’t dominate an entire field of research.
Mental Health in America
Mental health is both a monumental societal problem and a monumental scientific mystery. It’s one of the most important subjects that an agency like NIH could address.
As for the societal problem, many entire books have been written about this, so I’ll have to limit myself to a few datapoints about the national picture.
For an initial data point, suicide is one of the leading causes of death in the United States. The CDC reports that there were 45,979 suicides in 2020. But that just scratches the surface: that same year, “an estimated 12.2 million American adults seriously thought about suicide, 3.2 million planned a suicide attempt, and 1.2 million attempted suicide.”[7]
Just going by suicidal tendencies alone, the U.S. isn’t doing so well. But mental health can go wrong in many other ways. As of 2020, around 21 million US adults had major depression,[8] while over 10 percent of children are diagnosed with ADHD.[9]
The most recent news continues to be more and more disturbing. In early 2023, the CDC released a report that got headlines and attention everywhere, i.e., the most recent results from a survey titled “Young Risk Behavior Survey Data Summary & Trends Report: 2011-2021.”[10] That report found that more than 40% of high school students feel “so sad or hopeless that they could not engage in their regular activities for at least two weeks during the previous year.” Alarmingly, it found that 57% of female students (up from 36% in 2011) had “persistent feelings of sadness or hopelessness.”[11]
If over half of teen girls have “persistent feelings of sadness or hopelessness,” that seems like a major societal crisis for the next generation. Indeed, just one chart[12]—about time spent with friends—captures a dramatic difference between today’s teens and teenagers in prior generations:
It seems obvious that many mental health issues won’t be solved just with more brain scans and genetic screenings. Leaving aside schizophrenia and similar disorders for the moment, the recent increase in depression and anxiety may well be a reflection of the fact that people have actual, legitimate reasons to be unhappy about their lives and their social arrangements.
Wherever that’s the case, brain scans and drugs would just be a band-aid trying to cover up the real problem.
***
To be sure, other disorders may be relatively rare, but can have dramatic ramifications for the individuals, their families, and the rest of society. For example, people with schizophrenia make up a huge component of the homeless population. It’s impossible to come up with nationally representative statistics (for obvious reasons), but over two dozen studies of homeless people estimate the rate of psychosis or schizophrenia as 21% and 10% respectively,[13] which at a minimum is probably 20 times as high as the rate in the general population.[14]
The criminal justice system unsurprisingly deals with an enormous number of folks who are struggling with mental health issues. By the most recent official estimate, about 43% of state prisoners “had a history of a mental health problem.”[15] Despite all of these serious mental health issues, we have been on a national crusade to shut down mental health institutions for the past several decades. As of 2020, there were only 14,187 mental health facilities of all kinds in the US.[16] Almost all of them are either outpatient or community health centers.
As a result, we don’t have the institutional capacity to deal with people who have severe mental health problems. Instead, they end up on the street and in jail.
Indeed, there’s a grim joke in the criminal justice community:
Q. “What’s the largest mental hospital in the US?”
A. “The Los Angeles County jail.”
Obviously, the Los Angeles County jail isn’t an actual mental hospital with personnel, policies, and practices created to address mental health issues. But that’s the point. It’s a default mental hospital, because it ends up handling (however inadequately) many people with severe mental illnesses who act out and have nowhere else to go. As one expert told NPR, “Many times individuals who really do require intensive psychiatric care find themselves homeless or more and more in prison. Much of our mental health care now for individuals with serious mental illness has been shifted to correctional facilities.”[17]
In the past, mental hospitals were often far from adequate. There were good reasons that people in the mid-20th century thought that mental institutions should be abolished or cut significantly. But in abolishing many mental hospitals, we ended up putting the same people in jail instead. However bad mental hospitals used to be, jails are probably worse.
In a short-sighted attempt to do good, we made people’s lives unequivocally worse. We need a better path for providing treatment to people who are, at least for some time, largely unable to function in society.
***
As for the scientific mystery of mental health, this New York Times quote is on point:
“There have been only two major drug discoveries in the field in the past century; lithium for the treatment of bipolar disorder in 1949 and Thorazine for the treatment of psychosis in 1950. Both discoveries were utter strokes of luck, and almost every major psychiatric drug introduced since has resulted from small changes to Thorazine. Scientists still do not know why any of these drugs actually work.”[18]
In a way, it isn’t surprising that we don’t know very much about mental health and psychiatry. The human brain is unfathomably complex: Some 86 billion neurons with 1,000 to 10,000 or more connections each, plus 36-39 billion glial cells, receiving inputs from 12 million olfactory receptor cells and 140 million retinal receptor cells (among other things). The length of the nerve fibers in your brain, if stretched end to end, would be enough to stretch around the Earth four times![19]
Put all this together, and the real question might be: How does that ever work at all? How do any of us function with some semblance of normality?
There are so many ways that the brain can go wrong, and so few of those ways that we even remotely understand, that mental health is probably one of the hardest scientific problems in existence.
***
Indeed, many experts have argued that we don’t really have a solid understanding of basic issues like: 1) which mental disorders actually exist, 2) how to objectively diagnose them, 3) when and how multiple disorders occur simultaneously, and the like.
Not that there isn’t a ton of scholarship relating to these questions, to be sure! But a traditional tool of psychiatry—the Diagnostic and Statistical Manual of Mental Disorders (or DSM)—is widely acknowledged as deeply flawed in the way it draws up lists of (mostly self-reported) symptoms as to many different possible disorders.
For example, the current DSM lists many different types of anxiety-related disorders, along with diagnostic criteria, as if they are all separate conditions: “Separation anxiety disorder, selective mutism, specific phobia, social anxiety disorder, panic disorder, panic attack specifier, agoraphobia, generalized anxiety disorder, substance/medication-induced anxiety disorder, anxiety disorder due to another medical condition, other specified anxiety disorder, and unspecified anxiety disorder.”[20]
But one recent study found that mental disorders aren’t so easily demarcated: in a database of nearly 6 million medical records from Denmark, “every single mental disorder predisposed the patient to every other mental disorder—no matter how distinct the symptoms.”[21] Due to this and other research, a news article in Nature concluded that “the idea that mental illness can be classified into distinct, discrete categories such as ‘anxiety’ or ‘psychosis’ has been disproved to a large extent.”[22]
In other words, it may not make sense to try to classify all possible mental disorders (let alone 12 different types of anxiety) by a list of their symptoms, as if each is a distinct and stable disease. As Steve Hyman of Harvard (a former director of NIMH who actually helped draft the DSM) told Nature, “Any clinician could have told you that patients had not read the DSM and didn’t conform to the DSM.”
And as one scholar put it, an even more fundamental problem is that “to attempt to diagnose illness using patient symptoms resembles the approach of the eighteenth, not twenty-first-century medicine.”[23] For example, cardiologists have better ways of distinguishing between mild heart attacks and indigestion without being stuck with the observation that “the patient complains of chest pain.”
It should therefore be little surprise that when one set of scholars reviewed 102 meta-analyses covering 3,782 RCTs in mental health, they found that the risk of bias was high, and that there were “small benefits overall” for both psychotherapy and pharmaceuticals,. They concluded that “improving treatment strategies for mental disorders can be regarded as a central health challenge of the 21st century.”[24]
A final introductory note: drug development for mental health is extraordinarily hard.
Dr. William Potter, who was once a top researcher at the mental health institute and retired last year as the vice president of translational neuroscience at the giant drug maker Merck, said that far more basic research needed to be done on the causes of mental illness before anyone — industry or government — could successfully create breakthrough drugs. “We still don’t even understand how lithium works,” Dr. Potter said. “So how do people think we can find drugs systematically for mental illness?”[25]
With that as the depressing backdrop, here’s a short overview of the history of US government funding for mental health research:
The National Institute of Mental Health is one of the oldest NIH Institutes—it was officially launched in 1949, as called for by the National Mental Health Act of 1946 (signed by President Truman).[26] Due to the randomness of politics, NIMH has taken many forms over the decades. For example, in 1967, it was split off from NIH, and as of 1968, it became part of a larger agency that no longer exists (the Health Services and Mental Health Administration).[27] In 1972, the National Institute on Drug Abuse was launched as part of NIMH, and in 1973, NIMH “temporarily rejoined NIH,” but then yet another agency was created (the Alcohol, Drug Abuse, and Mental Health Administration) that would be composed of NIMH, NIDA, and the National Institute on Alcohol Abuse and Alcoholism. That lasted until 1989, when Congress abolished that overarching agency, and as of 1992, it gave the research components of NIMH, NIDA, and NIAAA back to NIH, although the treatment programs were split off into yet another agency (Substance Abuse and Mental Health Services Administration).
Whew! And that wasn’t even all of it.
Indeed, the tortuous history of NIMH might help explain why this book doesn’t usually delve into all the administrative history of the rest of NIH. It can all be a bit mind-boggling, and only serves to prove that politicians and experts need several swings at the bat in order to get it right (if they ever do).
In 2002, Tom Insel was named director of NIMH. As a bit of background, he got a medical degree from Boston University, and then did clinical training in psychiatry at UCSF from 1976 to 1979. From there, he went to work at NIMH as a clinical fellow. After many years there, he went to Emory University in Atlanta in 1994 to direct a research center on primates, and in 1999, he began leading a major NSF-sponsored Center for Behavioral Neuroscience.[28] That was his final appointment before becoming NIMH Director in 2002.
Insel ended up taking NIMH in a very particular direction as to what was and wasn’t fundable. He preferred to fund studies involving neuroimaging and/or genetics, in an attempt to uncover the actual biological mechanisms in the brain that may underlie various mental illnesses.
Indeed, in 2010-11, NIMH announced (with some justification) that it was unhappy with the DSM and the traditional ways of diagnosing mental illness. As Steve Hyman (himself a former NIMH director) told Science, the current DSM model of diagnosing mental illness made little sense. For example, a diagnosis of major depression could be made if a patient had at least five of nine symptoms. But “in this scenario, it’s possible for two patients to receive the same diagnosis with only one symptom in common.”[29] Such a system of diagnosing disease struck many people as less than objective or satisfactory.
NIMH therefore launched an approach called Research Domain Criteria (RDoC) as a replacement for the traditional way of classifying and diagnosing mental disorders. As an NIMH researcher told Science in 2010, “What we are doing is trying to develop new ways to classify disorders that are based on identifiable neural circuits.”[30] The intent was to divide up psychiatry into five broad domains that are “present in everyone but whose extremes correspond to mental illness: negative emotionality, positive emotionality, cognitive processes, social processes, and arousal/regulatory systems.”[31] Then, in 2012, Insel co-authored a scholarly article arguing that psychiatric diagnoses had no “biological ‘gold standard’ definition,” that we are left with “a profusion of statistically significant, but minimally differentiating, biological findings,” and that rather than sticking with the DSM-listed disorders, we should focus on developing new ways to identify the underlying biological differences.[32]
By 2013, Insel drew attention—not all of it positive—by writing a blog post formally announcing that NIMH’s funding would no longer look to the American Psychiatric Association’s famous DSM.[33] In his words, the “weakness” of DSM is its “lack of validity.” Unlike the way we define heart disease, cancer, or AIDS, “the DSM diagnoses are based on a consensus about clusters of clinical symptoms, not any objective laboratory measure. In the rest of medicine, this would be equivalent to creating diagnostic systems based on the nature of chest pain or the quality of fever.” He then announced that NIH’s RDoC project (“Research Domain Criteria”) would “transform diagnosis” by including “genetics, imaging, cognitive science,” and more.
As the New York Times reported, “Just weeks before the long-awaited publication of a new edition of the so-called bible of mental disorders, the federal government’s most prominent psychiatric expert has said the book suffers from a scientific ‘lack of validity.’”[34] The Times quoted Insel as saying: “As long as the research community takes the D.S.M. to be a bible, we’ll never make progress. People think that everything has to match D.S.M. criteria, but you know what? Biology never read that book.”
At the same time, the Times itself was well aware of the many difficulties at hand:
Decades of spending on neuroscience have taught scientists mostly what they do not know, undermining some of their most elemental assumptions. Genetic glitches that appear to increase the risk of schizophrenia in one person may predispose others to autism-like symptoms, or bipolar disorder. The mechanisms of the field’s most commonly used drugs — antidepressants like Prozac, and antipsychosis medications like Zyprexa — have revealed nothing about the causes of those disorders. And major drugmakers have scaled back psychiatric drug development, having virtually no new biological “targets” to shoot for.[35]
Insel’s blog post was widely criticized, and not just by psychiatrists who thought their main body of work had been insulted. Dorothy Bishop at Oxford, for example, saw “big problems” with the new NIMH approach.[36] For example, the NIMH’s approach failed to include “anything about experience or environment,” and instead assumed that all mental problems are “disorders of brain circuits.” Relatedly, she thought there was no evidence that “better knowledge of neurobiological correlates” would help improve psychological interventions for, say, obsessive-compulsive disorder. For another example, the NIMH proposal took a “naïve view of the potential of genetics” to improve psychiatry. Most behaviors have, at most, a very tiny association with any gene, and pouring money into genetics at present “sounds to me like a recipe for wasting a huge amount of research funding.”
Bishop was far from alone in her skepticism about NIMH’s turn towards genetics and imaging. As Nature pointed out, “clinical researchers bristled in 2013 when former NIMH director Thomas Insel announced that the agency would shift away from funding research that classified people using DSM-5, and again in 2014 when Insel said that the NIMH would not fund clinical trials that didn’t seek to understand the biological mechanism underlying a particular treatment or illness.”[37]
The New York Times wrote in 2014 that Insel’s tenure at NIMH created a “departure” that was “far larger than just about anyone could have anticipated.” He “sharply shifted the agency’s focus—to basic neuroscience and genetics, at the expense of the very type of behavioral research he himself had once done.”[38] He became convinced that “the only way to build a real psychiatric science is from first principles — from genes and brain biology, as opposed to identifying symptom clusters. Some of the mental health institute’s largest outlays under Dr. Insel have been to support projects that, biologically speaking, are like mapping the ocean floor.”[39] As one respected researcher from Duke told the Times, “N.I.M.H. is betting the house on the long shot that neuroscience will come up with answers to help people with serious mental illness. . . . It does little or no psychosocial or health services research that might relieve the current suffering of patients.”[40]
How did NIMH’s priorities arguably distort research? One notable example: Kristina Olson was a psychology professor at Princeton who studied gender and sex issues. She wrote that she and her colleagues “were so desperate for funding for our trans work” that they considered flying patients “to Seattle to put them in a scanner, [and] then get out to do the actual study we thought mattered (but wasn’t getting funded). That is how crazy this is. It would have added a million $ & been more fundable.”[41]
In other words, she couldn’t get her actual study funded, but thinks that if she had added a million dollars in funding so as to fly people around the country to have their brains scanned, that would have made NIMH under Tom Insel more likely to fund her work.
Here’s another example of how NIMH’s approach was arguably too narrow. NIMH issued a funding announcement on eating disorders in 2013. It stated that the primary goals were to:
“(1) support integrative, hypothesis-driven studies of neural circuits and/or other biological mechanisms underlying eating disorders; (2) support the use of dimensional constructs (defined for the purposes of this FOA below) as a primary means to investigate these mechanisms; (3) support the delineation of trajectories over time (e.g., across developmental stages or across illness course); (4) encourage integration across different levels of analysis (e.g., behavior, cells, circuits, genes, molecules, physiology, self-report, symptoms); (5) encourage neurodevelopmental research in eating disorders; and (6) encourage application of systems neuroscience methods to the study of eating disorders.”[42]
That’s fairly dense. What does it all mean?
As psychology professor Sanjay Srivastava wrote at the time,[43] those funding priorities were all about neuroscience, genes, biological mechanisms, etc. But if you actually want to address eating disorders, some important words and concepts were entirely missing: “social; media; culture; family; peer (when not followed by ‘review’ referring to the funding processes); body image; self.”
As he put it, “if NIMH thinks that basic research on media, on family environments, on peer influence, on self-concept, on cultural norms are not terribly important for understanding and treating eating disorders — well, that’s really hard to defend.”
As one might suspect, one fairly dramatic change at NIMH over these years was that it stopped supporting as many clinical trials on mental disorders. One analysis by Nature in 2017 found that “the number of clinical trials funded by the National Institute of Mental Health has fallen by 45% since the agency began to focus on the biological roots of disease.”[44] As that article pointed out, “The NIMH’s embrace of fundamental research has infuriated many clinical researchers, who see it as an attempt to invalidate their methods — and say that there is scant evidence to support the idea that using RDoC will lead to greater insight or better treatments for mental illness.” As Dr. Allen Frances from Duke told the New York Times, “Instead of being an institute of mental health, he has made it almost exclusively a brain research institute.”[45] Another review article found that “between 2006 and 2023, NIMH-funded extramural drug trials for schizophrenia and bipolar disorder decreased by 95%”![46]
As well, NIMH seemed to lean into the trend of launching well-funded initiatives or moonshots to much fanfare and publicity. The results were often impressive, but often criticized as well, and it’s not clear how much they have yet contributed to improving mental health in the real world.
One major effort from NIMH at the time was the Psychiatric Genomics Consortium.[47] The claim was that “by bringing together genetic data from hundreds of thousands of individuals around the world, the PGC is working to find the genetic variants that change risk of disease. Identifying the genetic basis of these disorders helps us to understand the underlying biology and to develop better ways of preventing and treating disorders.”[48] This consortium seems to be widely accepted as a success, at least as to genetic data, but as of yet there is no evidence that they have improved anything for patients.
Another major effort – the Human Connectome Project – was “launched in 2009” as an “ambitious effort to map the neural pathways that underlie human brain function.”[49] It officially got to work in 2010 when it “awarded $40 million to two collaborating research consortia to map the human brain’s connections in high resolution.” The first consortium (Minnesota, Oxford, and Washington University in St. Louis) “set out to comprehensively map human brain circuitry in 1200 healthy adults using cutting-edge methods of noninvasive neuroimaging.” The other consortium (UCLA, MGH/Harvard) tried to “create a new magnetic resonance imager optimized for measuring connectome data.”[50]
The results? A 2021 article by the project leaders noted that “more than 27 Petabytes of data have been shared, and 1,508 papers acknowledging HCP data use have been published.”[51] The article also noted “several scientific advances,” including “improved cortical parcellations, analyses of connectivity based on functional and diffusion MRI, and analyses of brain-behavior relationships.” Even accepting all of these papers at face value, I don’t see much here that actually improves the situation for US mental health, and while I reached out to study leaders for comment, they did not respond with any explanation as to how all of this MRI-type research could, even in theory, directly solve mental health problems.
Then, in 2015, NIMH and a few other Institutes and Centers joined to fund the so-called “ABCD Consortium,” which would scan the brains of over 10,000 children, and follow them into adulthood. At a cost of $300 million, the study will track quite a number of things, including “how childhood experiences (such as sports, videogames, social media, unhealthy sleep patterns, and smoking) interact with each other and with a child’s changing biology to affect brain development and social, behavioral, academic, health, and other outcomes.”[52] A New York Times article dismissively said that there are “so many interacting variables of experience and development that it’s hard to discern what the study’s primary goals are.”[53]
As well, the NIH has a $50 million project called the PsychENCODE project that is a “collaboration between 15 research institutes working to provide an enhanced framework of regulatory genomic elements in individuals with neuropsychiatric disorders.”[54] What does this mean? In large part, scanning over 2,000 brains to look for genetic links to things like schizophrenia, autism, and bipolar disorder. A critic of the project (Dr. Dan Graur, an evolutionary geneticist in Texas) said, “If you take something vague [i.e., a diagnosis of schizophrenia] and correlate it with millions of genetic and epigenetic variations, you are bound to get statistical significance that will have little biological significance.”[55] As the Times noted, the prominent psychological researcher Scott Lilienfeld said of such projects, “They’re either fishing expeditions or Hail Marys. Take your pick.”[56]
There is good reason to be skeptical that neuroimaging is anywhere near being able to help with clinical approaches. One paper in Nature recently found that brain-wide association studies (where the researchers try to associate cognitive or psychiatric characteristics with brain imaging data) are typically far too small, with a median sample size of 25 even though you would need thousands of participants to get accurate results.[57]
To make matters worse, another recent paper found that current machine learning approaches can’t diagnose major depressive disorder from neuroimaging data. Specifically, the study looked at 1,801 patients with major depression, and compared them to healthy patients based on detailed neuroimaging data.
The results: Even with 2.4 million machine learning models, the accuracy of predicting a depression diagnosis never rose about 62%, not much better than flipping a coin. The conclusion: “although multivariate neuroimaging markers increase predictive performance compared to univariate analyses, classification on the level of the individual patient—even under optimal conditions—does not reach clinically relevant levels.”[58]
Keep in mind that this study was larger and more sophisticated than most brain imaging studies—as Dorothy Bishop of Oxford said to me, maybe we should not be funding those smaller studies in the first place.
The story of another paper is remarkable. In 2017, Nature Human Behaviour published what seemed to be a blockbuster paper (sponsored in part by an NIMH grant).[59] The paper claimed that when you put suicidal youth into an fMRI machine and looked at neural representations associated with concepts like “death,” “cruelty,” “trouble,” “carefree,” “good,” and “praise,” a machine learning algorithm was able to distinguish them from healthy control subjects with 91% accuracy, and that an algorithm could even tell with 94% accuracy who had actually tried to commit suicide versus who had not. As neuroscientist Konrad Kording said, “If true, the paper’s approach could revolutionize psychiatric approaches to suicide.”[60]
But it wasn’t true, as Kording himself and a colleague showed. A full six years later in 2023—after several years of effort—the paper was retracted. The paper—which had only 17 suicidal subjects and 17 controls—was full of overfitting. A rigorous way to do machine learning on medical data (or any other data, for that matter) is to divide up the dataset ahead of time, and to reserve some of the data (called a “test set”) to be used only at the very end of the study, as a final test of whether the algorithm you created on the other data is usable on data that is still fresh, so to speak. But here, as Kording said, “The authors apparently used the test data to select features. Obvious mistake.”[61]
The result: Overfitting. It’s a little bit analogous to shooting an arrow into a wall, and then painting a target around where the arrow landed. You can’t use where the arrow landed to pick where the target should be. That’s backward. The target has to be picked ahead of time, and then tested fairly.
The problem with neuroscience may go much deeper, however. As neuroscientist Erik Hoel points out in his book The World Behind the World, we don’t even know that average brain activity (which is behind “almost all the reported effects in neuroscience”) is a meaningful thing to measure and interpret. He points to animal studies finding that an individual animal’s brain activity and behavior didn’t necessarily correspond either with average brain activity or average behavior, indicating “that the statistical constructs used to create neuroscience papers are epiphenomena for the brain itself—as if we are trying to understand a clock by the shadows it casts.”[62]
Another classic article in neuroscience showed that if you analyze a microprocessor by limiting yourself to the typical neuroscience techniques, you’d never come close to understanding the inner workings of the microprocessor. This “suggests that the availability of unlimited data, as we have for the processor, is in no way sufficient to allow a real understanding of the brain.”[63]
In other words, while we should fund some neuroimaging and machine learning studies, we shouldn’t put all our eggs in that basket while millions of people are struggling with mental health.
The same may be true of genetics. Despite the amazing advances in that field in the past 20-25 years, one recent article pointed out that “it is still not possible to cite a single neuroscience or genetic finding that has been of use to the practicing psychiatrist in managing these illnesses despite attempts to suggest the contrary.”[64] “Rather than neuroscience research, serendipity or lateral thinking remain the key tools in psychotropic drug discovery.” The article goes on to make the common sense point that we ought to diversify our mental health funding: “Investment in psychological, social science and service delivery research has occurred but represents only a small fraction of the attention and funding of neuroscience. . . . Is it not time for research to refocus resource and expertise away from the laboratory and onto these more relevant psychological and social sciences and research into clinical practice, public health and service delivery?”
You might think that the scholar here might have overstated things in saying that nothing from neuroscience or genetics has proven useful to the practicing psychiatrist. And a commentary on that article does make that counterargument: “The most recent genome-wide association study on depression found 87 independent loci that were associated with depression, with a startling lack of genes involved in the 5-HT system. . . . Findings such as these are likely to be of great benefit in developing new treatments.”[65]
Maybe so. But if you follow the reference for that claim, you end up with an article that performed a heroic amount of work—analyzing data on 807,553 individuals from the largest genome-wide association studies on depression, along with an ”independent replication sample of 1,306,354 individuals,” in order to narrow in on 87 genetic variants, all of which put together explained only 0.8% to 3.2% of the variance in depression.[66]
In other words, the scholars studied thousands of genes across over 2 million patients, and in the end were only able to find 87 genes that collectively explain at most about 3% of why someone gets depressed (and nothing about the remedy, because this study didn’t even look at that). While an impressive achievement, this sort of study is still nowhere near being able to give specific clinical advice on how psychiatrists should treat an actual patient with depression.
In short, we need mental health funding that isn’t consumed with the latest fad (genetics, machine learning, the microbiome, or whatever). Mental health is an incredibly complex problem that needs attention at many different levels—yes, genetics and machine learning, but also community-based interventions, and therapeutic programs, and more. The most complex problem in the universe doesn’t need the most simplistic approach.
Even now, several years after Insel’s departure from NIMH, researchers are still feeling the impact. One neuroscientist from Princeton recently wrote that funding priorities at NIMH are the “exact opposite” from what we need to actually understand mental health:
“As examples, a program officer at the National Institute for Mental Health (NIMH) requested that a colleague withdraw her funding application from consideration prior to review, saying that regardless of reviewers’ evaluation, the clearly mental-health relevant research will not be funded as it does not include a neural component; I have similarly been told, on consultation with several program officers, that a computational psychiatry center that focuses on behavioral measures is not of interest to NIMH unless we include neuroscience methods such as fMRI or MEG, despite research so far showing little return for such techniques in understanding mental illness (Roiser, 2015), not to mention in developing clinically-feasible tools for diagnosis and treatment selection.”[67]
As a number of prominent psychiatrists who had served on NIMH’s Advisory Council wrote in 2016, “diversification is a prudent strategy” in any investment portfolio, and a “disproportionate investment in neuroscience is as imprudent as investing only in growth stocks and neglecting less risky investments that yield immediate albeit potentially more modest benefits.”[68] And yet, at the time, basic and translational neuroscience research made up some 85% of NIMH’s funding, with only 15% going towards services, programs, and interventions.
From a structural perspective, it seems like a bad idea to give an NIH Institute Director the power to make radical changes in any one direction, turning the entirety of a given scientific field into his or her personal plaything.
Sure, a Feynman or Einstein or Curie might be able to wield that power wisely, but the vast majority of people won’t. By far, most scientists (especially the type of person who often rises to the top of a government bureaucracy) should be more prudent and humble about whether their own personal views should dominate an entire field.
NIH should give more thought towards preventing what happened at NIMH under Tom Insel. Perhaps NIH should engage in “red teaming” exercises (a term from military and intelligence sources, where a team of people is deliberately tasked with trying to poke holes and critique the current approach).
As well, NIH should systematically set aside 20% or more of its funding to ideas, theories, and lines of research that aren’t currently in favor. At the other end of the spectrum, NIH should set aside 20% of funding for replication studies and methods development/evaluation so that we know what is actually reliable in this area.
Nothing in the history of science would make us think that it’s a good idea for one person or one idea to gain such prominence that they are able to squelch other approaches. And as shown in other chapters, the structure and model of NIH (with a fairly rigid system of peer review) is biased towards creating groupthink, rather than combatting it.
[1] Adam Rogers, “Star Neuroscientist Tom Insel Leaves the Goodle-Spawned Verily for . . . a Startup?,” Wired (May 11, 2017), available at https://www.wired.com/2017/05/star-neuroscientist-tom-insel-leaves-google-spawned-verily-startup/.
[2] Benedict Carey, “Science Plays the Long Game. But People Have Mental Health Issues Now,” New York Times (April 1, 2021), available at https://www.nytimes.com/2021/04/01/health/mental-health-treatments.html.
[3] See his tweet of Feb. 22, 2022, at
Josh Dubnau 🇵🇸@joshdubnau
@ent3c Many of us have been crying foul for years.
.
[4] See his tweet of Feb. 22, 2022, at
https://twitter.com/bradpwyble/status/1496334071315714048
.
[5] See his tweet of Feb. 22, 2022, at
https://twitter.com/bradpwyble/status/1496334071315714048
.
[6] See his tweet of Feb. 22, 2022, at
John W. Krakauer@blamlab
Yes – the neglect of behavioral research is a source of great shame. This belief in genes and magic bullets is a cult. t.co/nEJ8XVHRMR
.
[7] See https://www.cdc.gov/suicide/facts/index.html.
[8] See https://www.nimh.nih.gov/health/statistics/major-depression.
[9] See https://www.nimh.nih.gov/health/statistics/attention-deficit-hyperactivity-disorder-adhd.
[10] See https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBS_Data-Summary-Trends_Report2023_508.pdf.
[11] https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBS_Data-Summary-Trends_Report2023_508.pdf#page=64.
[12] See
The new CDC report shows that Covid added little to teen mental health trends
For the first few months of this new substack, I plan to publish a major post every two or three weeks. In the weeks in between, like today, I’ll sometimes write something shorter and respond to the best criticisms of my previous post. I’ll usually write these responses with Zach Rausch, the lead researcher for this substack…
Read more
3 years ago · 169 likes · 87 comments · Jon Haidt
.
[13] Getinet Ayano, Getachew Tesfaw, and Shegaye Shumet, “The prevalence of schizophrenia and other psychotic disorders among homeless people: a systematic review and meta-analysis,” BMC Psychiatry 19 (2019), available at https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-019-2361-7.
[14] See https://www.nimh.nih.gov/health/statistics/schizophrenia.
[15] Bureau of Justice Statistics, Indicators of Mental Health Problems Reported by Prisoners: Survey of Prison Inmates, 2016 (published in 2021), available at https://bjs.ojp.gov/library/publications/indicators-mental-health-problems-reported-prisoners-survey-prison-inmates.
[16] See the National Mental Health Services Survey: 2020, available at https://www.samhsa.gov/data/sites/default/files/reports/rpt35336/2020_NMHSS_finsal.pdf, p. 9, Table 1.1.
[17] Samantha Raphelson, “How The Loss Of U.S. Psychiatric Hospitals Led To A Mental Health Crisis,” NPR (Nov. 30, 2017), available at https://www.npr.org/2017/11/30/567477160/how-the-loss-of-u-s-psychiatric-hospitals-led-to-a-mental-health-crisis.
[18] See https://www.nytimes.com/2011/01/23/health/policy/23drug.html?pagewanted=all.
[19] See Bente Pakkenberg et al., “Aging and the human neocortex,” Experimental Gerontology 38 (2003): 95-99.
[20] See https://dsm.psychiatryonline.org/doi/full/10.1176/appi.books.9780890425787.x05_Anxiety_Disorders.
[21] Michael Marshall, “The hidden links between mental disorders,” Nature (May 5, 2020), available at https://www.nature.com/articles/d41586-020-00922-8.
[22] Michael Marshall, “The hidden links between mental disorders,” Nature (May 5, 2020), available at https://www.nature.com/articles/d41586-020-00922-8.
[23] Andrew Scull, “American psychiatry in the new millennium: a critical appraisal,” Psychological Medicine (2021): 1-9.
[24] Falk Leichsenring et al., “The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: an umbrella review and meta-analytic evaluation of recent meta-analyses,” World Psychiatry (Jan. 11, 2022), available at https://onlinelibrary.wiley.com/doi/10.1002/wps.20941.
[25] See https://www.nytimes.com/2011/01/23/health/policy/23drug.html?pagewanted=all.
[26] See https://web.archive.org/web/20070310071549/https://www.nih.gov/about/almanac/archive/1999/organization/nimh/history.html. See also https://govtrackus.s3.amazonaws.com/legislink/pdf/stat/60/STATUTE-60-Pg421.pdf.
[27] See https://web.archive.org/web/20070310071549/https://www.nih.gov/about/almanac/archive/1999/organization/nimh/history.html.
[28] See https://en.wikipedia.org/wiki/Thomas_R._Insel.
[29] Greg Miller, “Beyond DSM: Seeking a Brain-Based Classification of Mental Illness,” Science327 no. 5972 (2010): 1437.
[30] Greg Miller, “Beyond DSM: Seeking a Brain-Based Classification of Mental Illness,” Science327 no. 5972 (2010): 1437.
[31] Greg Miller, “Beyond DSM: Seeking a Brain-Based Classification of Mental Illness,” Science327 no. 5972 (2010): 1437.
[32] S. Kapur, A. G. Phillips, & T. R. Insel, “Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it?,” Molecular Psychiatry 17 (2012): 1174-79.
[33] An archived version of his blog post is here: https://web.archive.org/web/20130503094041/http://www.nimh.nih.gov/about/director/2013/transforming-diagnosis.shtml.
[34] Pam Belluck and Benedict Carey, “Psychiatry’s Guide Is Out of Touch With Science, Experts Say,” New York Times (May 6, 2013), available at https://www.nytimes.com/2013/05/07/health/psychiatrys-new-guide-falls-short-experts-say.html?pagewanted=all.
[35] Pam Belluck and Benedict Carey, “Psychiatry’s Guide Is Out of Touch With Science, Experts Say,” New York Times (May 6, 2013), available at https://www.nytimes.com/2013/05/07/health/psychiatrys-new-guide-falls-short-experts-say.html?pagewanted=all.
[36] See http://deevybee.blogspot.com/2014/05/changing-landscape-of-psychiatric.html.
[37] Sara Reardon, “US mental-health agency’s push for basic research has slashed support for clinical trials,” Nature 546, 339 (2017), available at https://www.nature.com/articles/546338a.
[38] Benedict Carey, “Blazing Trails in Brain Science,” New York Times (Feb. 3, 2014), available at https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.
[39] Benedict Carey, “Blazing Trails in Brain Science,” New York Times (Feb. 3, 2014), available at https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.
[40] Benedict Carey, “Blazing Trails in Brain Science,” New York Times (Feb. 3, 2014), available at https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.
[41] See her tweet of April 2, 2021, at
https://twitter.com/olsonista/status/1377981103592574992
.
[42] See https://grants.nih.gov/grants/guide/rfa-files/RFA-MH-14-030.html.
[43] See https://thehardestscience.com/2013/05/24/where-is-rdoc-headed-a-look-at-the-eating-disorders-foa/.
[44] Sara Reardon, “US mental-health agency’s push for basic research has slashed support for clinical trials,” Nature 546, 339 (2017), available at https://www.nature.com/articles/546338a.
[45] See https://www.nytimes.com/2014/02/04/science/blazing-trails-in-brain-science.html.
[46] See E. Fuller Torrey, Wendy Simmons, and Lisa Dailey, “Schizophrenia, clinical and basic research at NIMH: A 75 Year retrospective,” Psychiatry Research 342 (Dec. 2024), available at https://www.sciencedirect.com/science/article/abs/pii/S0165178124005109?dgcid=coauthor.
[47] See
.
[48] See https://pgc.unc.edu/for-the-public/.
[49] See https://neuroscienceblueprint.nih.gov/human-connectome/connectome-programs.
[50] See https://neuroscienceblueprint.nih.gov/human-connectome/connectome-programs.
[51] See Jennifer Stine Elam et al., “The Human Connectome Project: A Retrospective,” Neuroimage 244 (2021), available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9387634/.
[52] See https://abcdstudy.org/about/.
[53] Benedict Carey, “Science Plays the Long Game. But People Have Mental Health Issues Now,” New York Times (April 1, 2021), available at https://www.nytimes.com/2021/04/01/health/mental-health-treatments.html.
[54] See
http://www.psychencode.org/
.
[55] See Kelly Servick, “Genomic data from 2000 human brains could reveal roots of schizophrenia, autism, and other neurological disorders,” Science (Dec. 13, 2018), available at https://www.sciencemag.org/news/2018/12/genomic-data-2000-human-brains-could-reveal-roots-schizophrenia-autism-and-other.
[56] Benedict Carey, “Science Plays the Long Game. But People Have Mental Health Issues Now,” New York Times (April 1, 2021), available at https://www.nytimes.com/2021/04/01/health/mental-health-treatments.html.
[57] Scott Marek et al., “Reproducible brain-wide association studies require thousands of individuals,” Nature 603 no. 7902 (2022): 654-660.
[58] Nils R. Winter et al., “A Systematic Evaluation of Machine Learning-based Biomarkers for Major
Depressive Disorder across Modalities” (Feb. 2023), preprint available at https://www.medrxiv.org/content/10.1101/2023.02.27.23286311v1.
[59] Marcel Adam Just et al., “Machine learning of neural representations of suicide and emotion concepts identifies suicidal youth,” Nature Human Behaviour 1 (2017): 911-919.
[60] See
Kording Lab 🦖@KordingLab
Here is the retracted paper: nature.com/articles/s4156… and here is our refutation nature.com/articles/s4156…. If true, the paper’s approach could revolutionize psychiatric approaches to suicide.
1:02 AM · Apr 7, 2023 · 14.8K Views
.
[61] See
Kording Lab 🦖@KordingLab
So what went wrong? The authors apparently used the test data to select features. Obvious mistake. A reminder for everyone into ML: never use the test set for *anything* but testing. Only practical way to do so in medicine? Lock away the test set till algorithm is registered.
1:02 AM · Apr 7, 2023 · 33.9K Views
6 Replies · 29 Reposts · 210 Likes
.
[62] Erik Hoel, The World Behind the World: Consciousness, Free Will, and the Limits of Science (2023), pp. 54-55.
[63] Eric Jonas and Konrad Paul Kording, “Could a Neuroscientist Understand a Microprocessor?,” PLoS Computational Biology (Jan. 12, 2017), available at https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268.
[64] David Kingdon, “Why hasn’t neuroscience delivered for psychiatry,” BJPsych Bulletin 44 no. 3 (Feb. 13, 2020): 107-09, available at https://www.cambridge.org/core/journals/bjpsych-bulletin/article/why-hasnt-neuroscience-delivered-for-psychiatry/2EB9F2202E61BCC98A5D1E5F5F825607.
[65] Lindsey Isla Sinclair, “What neuroscience has already done for us: Commentary on ‘Why hasn’t neuroscience delivered for psychiatry?,’” BJPsych Bulletin 44 no. 3 (June 2020): 110-112, available at https://www.cambridge.org/core/journals/bjpsych-bulletin/article/what-neuroscience-has-already-done-for-us/73A1986D1F5097D6555474ECAD8F63B3.
[66] See Table 2 in David M. Howard et al., “Genome-wide meta-analysis of depression identifies 102 independent variants and highlights the importance of the prefrontal brain regions,” Nature Neuroscience 22 no. 3 (March 2019): 343-352, available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6522363/.
[67] Yael Niv, “The primacy of behavioral research for understanding the brain,” PsyArXiv preprint (2020), available at https://psyarxiv.com/y8mxe/. This was later published at Behavioral Neuroscience 135 no. 5 (2021): 601-609.
[68] Roberto Lewis-Fernandez, “Rethinking funding priorities in mental health research,” British Journal of Psychiatry 208 (2016): 507-09.