Richard Horton in the Lancet writes:
Imagine if the entire edifice of knowledge in medicine was built upon a falsehood. Systematic reviews are said to be the highest standard of evidence-based health care. Regularly updated to ensure that treatment decisions are built on the most up-to-date and reliable science, systematic reviews and meta-analyses are widely used to inform clinical guidelines and decision making. Powerful organisations have emerged to construct a knowledge base in medicine underpinned by the results of systematic reviews. One such organisation is Cochrane, with 11 000 members in over 130 countries. This extraordinary movement of people is justifiably passionate about the idea that it is contributing to better health outcomes for everyone, everywhere. The industry that drives the production of systematic reviews today is financed by some of the most influential agencies in medical research. Cochrane, for example, points to three funders providing over £1 million each—the UK’s National Institute for Health Research (NIHR), the US National Institutes of Health (NIH), and Australia’s National Health and Medical Research Council (NHMRC).
Well, it really is a bit late for all this soul searching. See my earlier post here ‘Mega-silliness’ (commenting on what others had already pointed out); or my Evidence Based Medicine: the Epistemology That Isn’t, written over 20 years ago; and my contribution to the wake (even if I didn’t put my hand in my pocket), Why we should let “evidence-based medicine” rest in peace. The genesis of EBM was as a cult whose foundational myth was that P values could act as a true machine. Those followers who had originally hoped for a place in the promised afterlife, soon settled for paying the bills, and EBM morphed into a career opportunity for those who found accountancy too daring. So, pace John Mayall on Jazz Blues Fusion, don’t come here to listen to an old record. I promise.
I have spend a lot of time recently sifting through the detritus of a career. Finally — well, I hope, finally — I have managed to sort out my books. All neatly indexed in Delicious Library, and now for once the virtual location mirrors the physical location. For how long I do not know. Since I often buy books based on reviews, I used to put a copy of the review in with the book (a habit I have dropped but need to restart). I rediscovered this one by David Colquhoun (DC) reviewing ‘The Diet Delusion’ by Gary Taubes in the BMJ (with the unexpurgated text on his own web site).
I am a big fan of DC as he has lived though the rise and decline of much higher education in the UK. And he remains fearless and honest, qualities that are not always at the forefront of the modern university. Quoting the great Robert Merton he writes:
“The organization of science operates as a system of institutionalized vigilance, involving competitive cooperation. In such a system, scientists are at the ready to pick apart and assess each new claim to knowledge. This unending exchange of critical appraisal, of praise and punishment, is developed in science to a degree that makes the monitoring of children’s behavior by their parents seem little more than child’s play”.
“The institutionalized vigilance, “this unending exchange of critical judgment”, is nowhere to be found in the study of nutrition, chronic disease, and obesity, and it hasn’t been for decades.”
On Taubes and his (excellent book):
It took Taubes five years to write this book, and he has nothing to sell apart from his ideas. No wonder it is so much better than a scientist can produce. Such is the corruption of science by the cult of managerialism that no university would allow you to spend five years on a book
(as would be expected the BMJ omitted the punch line — they would, wouldn’t they?)
There is also a neat quote from Taubes in one of the comments on DC’s page from Beth@IDblog, one that I will try hard not to forget:
Taubes makes a point at the end of the Dartmouth medical grand rounds video that I think is important: “I’m not trying to convince you that it’s true, I’m trying to convince you that it should be taken seriously.”
This is an article by Philip Stark in Nature published awhile back. I like it.
In 1992, philosopher Karl Popper wrote: “Science may be described as the art of systematic oversimplification — the art of discerning what we may with advantage omit.” What may be omitted depends on the discipline.
You can say this another way: all experiments do violence to the natural world. We always want to cleave at the joints. But doing so may lead to error.
In 1992, philosopher Karl Popper wrote: “Science may be described as the art of systematic oversimplification — the art of discerning what we may with advantage omit.” What may be omitted depends on the discipline. Results that generalize to all universes (or perhaps do not even require a universe) are part of mathematics. Results that generalize to our Universe belong to physics. Results that generalize to all life on Earth underpin molecular biology. Results that generalize to all mice are murine biology. And results that hold only for a particular mouse in a particular lab in a particular experiment are arguably not science.
Science should be ‘show me’, not ‘trust me’; it should be ‘help me if you can’, not ‘catch me if you can’. If I publish an advertisement for my work (that is, a paper long on results but short on methods) and it’s wrong, that makes me untrustworthy. If I say: “here’s my work” and it’s wrong, I might have erred, but at least I am honest.
In medicine we have particular problems. Repeating experiments in model organisms is often possible whereas in man things are much harder. There is an awful lot of published medical research that is not a reliable guide to action.
Well, I doubt if any readers of these scribblings will be shocked. After all TIJABP. But this piece by the editor of PNAS wonders if the day of meaningful editing is over. I hope not. Looking backwards over my several hundred papers, the American Journal of Human Genetics was the most rigorous and did the most to improve our manuscript.
“Communication” remains in the vocabulary of scientific publishing—for example, as a category of manuscript (“Rapid Communications”) and as an element of a journal name (Nature Communications)—not as a vestigial remnant but as a vital part of the enterprise. The goal of communicating effectively is also why grammar, with its arcane, baffling, or even irritating “rules,” continues to matter. With the rise of digital publishing, attendant demands for economy and immediacy have diminished the role of copyeditor. The demands are particularly acute in journalism. As The New York Times editorial board member Lawrence Downs (4) lamented, “…in that world of the perpetual present tense—post it now, fix it later, update constantly—old-time, persnickety editing may be a luxury…. It will be an artisanal product, like monastery honey and wooden yachts.” Scientific publishing is catching up to journalism in this regard.
Being a renowned scientist doesn’t ensure success. On the same day that molecular biologist Carol Greider won a Nobel prize in 2009, she learnt that her recently submitted grant proposal had been rejected. “Even on the day when you win the Nobel prize,” she said in a 2017 graduation speech at Cold Spring Harbor Laboratory in New York, “sceptics may question whether you really know what you’re doing.”
There is an interesting review in the Economist of the ‘Great Pretender: The Undercover Mission that Changed out Understanding of Madness,’ written by Susan Cahalan. The book is the story of the American psychologist David Rosenhan who “recruited seven volunteers to join him in feigning mental illness, to expose what he called the ‘undoubtedly counter-therapeutic’ culture of his country’s psychiatry”.
Rosenthal’s studies are well known and were influential, and some might argue that may have had have a beneficial effect on subsequent patient care. The question is whether they were true. The review states:
in the end Rosenham emerges as an unpalatable symptom of a wider academic malaise”.
As for the ‘malaise’, the reviewer goes on:
Many of psychology’s most famous experiments have recently been discredited or devalued, the author notes. Immense significance has been attached to Stanley Milgram’s shock tests and Philip Zimbardo’s Stanford prison experiment, yet later re-runs have failed to reproduce their findings. As Ms Cahalan laments, the feverish reports on the undermining of such theories are a gift to people who would like to discredit science itself.
I have a few disjointed thoughts on this. There are plenty of other considered critiques of the excesses of modern medical psychiatry. Anthony Clare’s ‘Psychiatry in Dissent’ was for me the best introduction to psychiatry. And Stuart Sutherland’s “Breakdown’ was a blistering and highly readable attack on medical (in)competence as much as the subject itself (Sutherland was a leading experimental psychologist, and his account is autobiographical). And might the cross-country diagnostic criteria studies not have happened without Rosenham’s work?
As for undermining science (see the quote above), I think unreliable medical science is widespread, and possibly there is more of it than in many past periods. Simple repetition of experiments is important but not sufficient, and betrays a lack of of understanding of why some science is so powerful.
Science owes its success to its social organisation: conjectures and refutations, to use Popper’s terms, within a community. Just repeating an experiment under identical conditions is not sufficient. Rather you need to use the results of one experiment to inform the next, and with the accumulation of new results, you need to build a larger and larger edifice which whilst having greater explanatory power is more and more intolerant of errors at any level. Building large structures out of Lego only works because of the precision engineering of each of the component bricks. But any errors only become apparent when you add brick-on-brick. When a single investigator or group of investigators have skin in the game during this process — and where experimentation is possible — science is at its strongest (the critiques can of course come from anywhere).
An alternative process is when the results of a series of experiments are so precise and robust that everyday life confirms them: the lights go on when I click the switch. This harks back to the reporting of science as ‘demonstrations’.
By these two standards much medical science may be unreliable. First, because the fragmentation of enquiry discourages the creation of broad explanatory theories or tests of the underlying hypotheses. The ‘testing’ is more whether a publishable unit can be achieved rather than nature understood. Second, in many RCTs or technology assessments there is little theoretical framework on which to challenge nature. Nor can everyday practice act as the necessary feedback loop in the way the tight temporal relationship between flipping the switch and seeing the light turn on can.
Perhaps, perhaps not. But when and where is even more important.
Hailed as a maths prodigy at school, Shields accepted a junior position at Merrill Lynch after studying engineering, economics and management at Oxford University because the trading room floor offered him a thrilling, dynamic environment. He was not alone: of 120 engineers in his year group at university, Shields added, only five went into engineering.
I think we should be much more cautious in attempting to direct young people’s choices beyond providing them with an education. We should feel proud of their independence of mind, remembering that supply side factors will likely win out over central planning. It is the supply side that we need to deal with, not least Putts Law. The same applies to medicine.
This personal story is worth a read for other lessons, too.
“If biology is difficult, it is because of the bewildering number and variety of things one must hold in one’s head”.
John Maynard Smith (1977).
Leo Szilard recalled, that when he did physics he could lounge in the bath for hours and hours, just thinking. Once he moved into biology things were never the same: he was always having to get out to check some annoying fact. Dermatology is worse, trust me.
Terrific interview with Sydney Brenner about the second greatest scientific revolution of the 20th century.
I think it’s really hard to communicate that because I lived through the entire period from its very beginning, and it took on different forms as matters progressed. So it was, of course, wonderful. That’s what I tell students. The way to succeed is to get born at the right time and in the right place. If you can do that then you are bound to succeed. You have to be receptive and have some talent as well…
To have seen the development of a subject, which was looked upon with disdain by the establishment from the very start, actually become the basis of our whole approach to biology today. That is something that was worth living for.
This goes for more than science and stretches out into far more mundane aspects of life. Is there any alternative?
Whatever the context(s) of these events, the words are right.
The job of a scientist is to look for the truth, and the job of a teacher is to help people to empower themselves. I failed to do my job on both counts.
It is however not just the job of scientists.
The quote below was from a piece in the Lancet by Richard Horton.
Reading [Bertrand]Russell today is a resonant experience. Existential fears surround us. Yet today seems a long way from the dream of Enlightenment. Modern science is a brutally competitive affair. It is driven by incentives to acquire money (research funding), priority (journal publication), and glory (prizes and honours). Science’s metrics of success embed these motivations deep in transnational scientific cultures. At The Lancet, while we resist the idea that Impact Factors measure our achievements, we are not naive enough to believe that authors do not judge us by those same numbers. It is hard not to capitulate to a narrow range of indicators that has come to define success and failure. Science, once a powerful force to overturn orthodoxy, has created its own orthodoxies that diminish the possibility of creative thought and experiment. At this moment of planetary jeopardy, perhaps it is time to rethink and restate the purpose of science.
I am just musing on this. We like to think that ‘freedom’ was necessary for a modern wealthy state. We are not so certain, now. We used to think that certain freedoms of expression underpinned the scientific revolution. We are having doubts about this, too. Maybe it is possible to have atom bombs and live in a cesspool of immorality. Oops…
Direct URL for this post.
It is said that much of the foundations of 20th century physics was done in coffee houses (or in the case of Richard Feynman in strip bars), but things were once done differently in the UK
With neither institutional nor government masters to answer to, the British cyberneticians were free to concentrate on what interested them. In 1949, in an attempt to develop a broader intellectual base, many of them formed an informal dining society called the Ratio Club. Pickering documents that the money spent on alcohol at the first meeting dwarfed that spent on food by nearly six to one — another indication of the cultural differences between the UK and US cyberneticians.
The work of the British pioneers was forgotten until the late 1980s when it was rediscovered by a new generation of researchers… A company that I cofounded has now sold more than five million domestic floor-cleaning robots, whose workings were inspired by Walter’s tortoises. It is a good example of how unsupported research, carried out by unconventional characters in spite of their institutions, can have a huge impact.
A review from 2010 by Rodney Brooks of MIT of “The Cybernetic Brain: Sketches of Another Future” in Nature (For more on Donald Michie and “in spite of their institutions” see here).
Direct URL for this post.
I have had of all people a historian tell me that science is a collection of facts, and his voice had not even the ironic rasp of one filing-cabinet reproving another.
Jacob Bronowski | Science and Human Values
Direct URL for this post.
I have removed the name of the institution only because so many queue to sell their vapourware in this manner
Precision Medicine is a revolution in healthcare. Our world-leading biomedical researchers are at the forefront of this revolution, developing new early diagnostics and treatments for chronic diseases including cancer, cardiovascular disease, diabetes, arthritis and stroke. Partnering with XXXXX, the University of XXXX has driven … vision in Precision Medicine, including the development of a shitload of infrastructure to support imaging, molecular pathology and precision medicine clinical trials…… XXXXXX is now one of the foremost locations in a three mile radius to pursue advances in Precision Medicine.
And He declared to them, “It is written: ‘My house will be called a house of prayer. But you are making it ‘a den of robbers.'” Matthew 21:13
Direct URL for this post.
Statistics — to paraphrase Homer Simpson’s thoughts on alcohol — is the cause of, and solution to, all of science’s problems.
Direct URL for this post.
Digging deep into some of my old notes, I came across this obituary of John Ziman written by Jerry Ravetz. I know both through their written work and was lucky enough to meet and chat briefly with John Ziman not long before he died. Ziman’s book “Real Science” is for me the classic account of what has happened to science as it moved from a ‘way of life’ to a job.
Jerry Ravetz writes:
I first became aware of him through his 1960 radio talk Scientists – Gentlemen Or Players?, where he observed how a career in science was starting to change, from being a vocation to being a job.
There was a paradox running through his later career, to which he must have been sensitive. He was a “Renaissance man” in a way highly desirable for a scientist, but he did not exert the influence that he might have hoped to. This was due less to the passion he deployed in argument than the times in which he found himself. The age of such eminent scientist-savants as JBS Haldane, JD Bernal and Joseph Needham was passing, while a new generation of socially responsible scientists had yet to establish itself. Those who reminded scientists of their social responsibilities were viewed with suspicion; and those who had stopped doing research were treated as defectors.
Direct URL for this post.
“It appears to me, the doing what little one can to encrease [sic] the general stock of knowledge is as respectable an object of life as one can in any likelihood pursue.”
Darwin. Letter to his sisters from the Beagle. Quoted in the London Review of Books 23-May-2019, Rosemary Hill.
Direct URL for this post.
In a 1963 letter to molecular biologist Max Perutz, he wrote, “It is now widely realized that nearly all the ‘classical’ problems of molecular biology have either been solved or will be solved in the next decade…The future of molecular biology lies in the extension of research to other fields of biology, notably development and the nervous system.”
Sydney observed, and predicted, the flow of science: “Progress depends on the interplay of techniques, discoveries, and ideas, probably in that order of decreasing importance,” he said.
Man, the toolmaker. In this particularly case, a very special one.
Sydney Brenner (1927–2019) | Science [Obit of Sydney Brenner]
Direct URL for this post.
Some traits, such as adult height, are readily measured. The heritability of this trait is ∼60 to 80%. Attempts to characterize “height genes” have resulted in the identification of tens of thousands of genes, each of which contributes a small amount to this heritability. The plethora of factors is almost inevitable, given the vast number of cellular and physiological steps involved in the development of an adult human being. A model that accounts for ∼40% of height variability predicts individual heights to within 4 cm for 50% of people, but with errors of more than 10 cm for 5%. Thus, a sophisticated genomic analysis can predict height to some extent, but not well enough for use in ordering tailored clothing. Most direct-to-consumer genomic results are based on much less detailed analyses and many involve complex traits, so considerable skepticism is appropriate.
But such sensible comments, will not stem the hype — or the investors.
Direct URL for this post.
Scope for recognizing and accommodating exceptional individuals has been diminishing in British universities ever since. Hamilton published relatively few papers, in generally low status journals, and gained only a handful of grants much later in life. Bureaucratic measures of performance are increasingly important and judge the impact of an article only by the journal it is published in. This seriously undervalues radical originality, which although extremely rare is utterly vital to science. It is disturbing that a young Bill Hamilton today would probably find an academic career even more difficult to pursue.
Alan Grafen, in his obituary of Bill Hamilton (Biogr. Mems Fell. R. Soc. Lond. 50, 109–132 (2004)).
I post this excerpt following a discussion with somebody who had never heard of him. Hamilton’s enormous contributions to biology are not well known. You also have to wonder if the lack of a Nobel for biology diminishes medicine in the long run. Some things do indeed get worse.
Direct URL for this post.
Sydney Brenner has died. Not quite the last of the handful of scientists who made one of the two scientific revolutions of the 20th century. The first half belonged to physics, the second to the biology that he co-created.
A precocious boy—a student at the University of the Witwatersrand by the time he was 15—and bullied for it, reading was his connection to the wider world. Courses, he said, never taught him anything. The way to learn was to get a book that told you how to do things, and then to start doing them, whether it was making dyes or, later in life, programming computers. If he thought more deeply than the other great biologists of his age, which he did, it was surely because he read further, too.
Reading Brenner was a staccato of insights. I hadn’t come across the ‘courses’ quote before, but no surprises there.
I like statistics and spent most of my intercalated degree ‘using’ medical stats (essentially, writing programs on an IBM 360 mainframe to handle a large dataset, that I could then interrogate using the GLIM package from the NAG). Yes, the days of batch processing and punchcards. I found — and still find — statistics remarkably hard.
I am always very wary of people who say they understand statistics. Let me rephrase that. I am very suspicious of non-professional statisticians who claim that they find statistics intuitive. I remember that it was said that even the great Paul Erdos got the Monty Hall problem wrong.
The following is from a recent article in Nature:
What will retiring statistical significance look like? We hope that methods sections and data tabulation will be more detailed and nuanced. Authors will emphasize their estimates and the uncertainty in them — for example, by explicitly discussing the lower and upper limits of their intervals. They will not rely on significance tests. When P values are reported, they will be given with sensible precision (for example, P = 0.021 or P = 0.13) — without adornments such as stars or letters to denote statistical significance and not as binary inequalities (P < 0.05 or P > 0.05). Decisions to interpret or to publish results will not be based on statistical thresholds. People will spend less time with statistical software, and more time thinking.
There is lots of blame to go around here. Bad teaching and bad supervision, are easy targets (too easy). I think there are (at least) three more fundamental problems.
Science has been thought of as a form of ‘reliable knowledge’. This form of words always sounded almost too modest to me, especially when you think how powerful science has been shown to be. But in medicine we are increasingly aware that much modern science is not a basis for honest action at all. Blake’s words were to the effect that ‘every honest man is a prophet’. I once miswrote this in an article I wrote as ‘every honest man is for profit’. Many an error….
That picture that changed everything. Nice piece in Nature tells the story. (Image: NASA)
In climate science, you can check out of the lab anytime you like, but you can never leave.
Dave Reay, University of Edinburgh, quoted in Nature this week.
This is from an interview with Geoffrey Hinton who — to paraphrase Peter Medawar’s comments about Jim Watson — has something to be clever about. The article is worth reading in full, but here are a few snippets.
Now if you send in a paper that has a radically new idea, there’s no chance in hell it will get accepted, because it’s going to get some junior reviewer who doesn’t understand it. Or it’s going to get a senior reviewer who’s trying to review too many papers and doesn’t understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that’s really bad…
What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That’s I think the main downside of the fact that we’ve got this inversion now, where you’ve got a few senior guys and a gazillion young guys.
I would make a few comments:
All has been said before, I know, but no apology will be forthcoming.
Genome-wide study of hair colour in UK Biobank explains most of the SNP heritability.
Michael D. Morgan, Erola Pairo-Castineira, Konrad Rawlik, Oriol Canela-Xandri, Jonathan Rees, David Sims, Albert Tenesa & Ian J. Jackson
[Link to Nature Comm paper] https://doi.org/10.1038/s41467-018-07691-z
My guess is this is likely my last ‘research paper’ (although I now choose to redefine what counts as research). But not my last ‘thinking paper’. I cannot help but contrast the sheer volume of activity with that from our original papers on red hair. Things seemed so much simpler when we were young. But it is a nice coda to a career fugue.
Leading universities should pledge to actually read the work of applicants for research positions rather than use controversial metrics during the selection process, a Nobel prizewinner has argued.
No, not a spoof, but words from Harold Varmus. Sydney Brenner, a good while back, observed that people tended not to read papers anymore, they just xeroxed them.
Modesty seems to be under negative selection — among modern scientists, at least. So I warmed to this comment on a report of some recent work on the genetics of Africa and hunter-gatherers.
Deepti Gurdasani, a genetic epidemiologist at the Wellcome Sanger Institute in Hinxton, UK. But it’s plausible, she adds. “There is literally nothing in Africa that is not possible since we have no idea what humans were doing on the continent 5,000 years ago.”
This is from an article in Nature.
Under pressure to turn out productive lab members quickly, many PhD programmes in the biomedical sciences have shortened their courses, squeezing out opportunities for putting research into its wider context. Consequently, most PhD curricula are unlikely to nurture the big thinkers and creative problem-solvers that society needs.
That means students are taught every detail of a microbe’s life cycle but little about the life scientific. They need to be taught to recognize how errors can occur. Trainees should evaluate case studies derived from flawed real research, or use interdisciplinary detective games to find logical fallacies in the literature. Above all, students must be shown the scientific process as it is — with its limitations and potential pitfalls as well as its fun side, such as serendipitous discoveries and hilarious blunders.
And from a letter in response
My father designed stellar-inertial guidance systems for reconnaissance aircraft and, after he retired, would often present his work to physics and engineering students. When they asked him what they should study to prepare for such a career, he would reply: “Read the classics,” by which he meant Aristotle, Ralph Waldo Emerson, Jean-Jacques Rousseau and Blaise Pascal.
The best scientific and technical progress does not come out of a box. It is more likely to emerge from trying to fit wild, woolly and tangential ideas into useful societal and economic contexts.
As the historian Norman Davies once said:
“Since no one is judged competent to offer an opinion beyond their own particular mineshaft, beasts of prey have been left to prowl across the prairie unchecked.”
Or as the Economist once put it”
“…professors fixated on crawling alone the frontiers of knowledge with a magnifying glass.”
This is the tragedy of our age: 90% right and 100% wrong. And that is even before we get to medicine.