Not all scientific studies are created equal – David H. Schwartz

Not all scientific studies are created equal – David H. Schwartz


Studies have shown that taking vitamins is good for your health and bad for your health. That newly discovered herb can improve your memory or destroy your liver. Headlines proclaim a promising new cancer treatment and never mention it again. On a daily basis, we are bombarded with attention-grabbing news, backed up by scientific studies, but what are these studies? How are they performed? And how do we know whether they’re reliable? When it comes to dietary or medical information, the first thing to remember is that while studies on animals or individual cells can point the way towards further research, the only way to know how something will affect humans is through a study involving human subjects. And when it comes to human studies, the scientific gold standard is the randomized clinical trial, or RCT. The key to RCTs is that the subjects are randomly assigned to their study groups. They are often blinded to make them more rigorous. This process attempts to ensure that the only difference between the groups is the one the researchers are attempting to study. For example, when testing a new headache medication, a large pool of people with headaches would be randomly divided into two groups, one receiving the medication and another receiving a placebo. With proper randomization, the only significant overall difference between the two groups will be whether or not they received the medication, rather than other differences that could affect results. Randomized clinical trials are incredible tools, and, in fact, the US Food and Drug Administration often requires at least two to be conducted before a new drug can be marketed. But the problem is that an RCT is not possible in many cases, either because it’s not practical or would require too many volunteers. In such cases, scientists use an epidemiological study, which simply observes people going about their usual behavior, rather than randomly assigning active participants to control invariable groups. Let’s say we wanted to study whether an herbal ingredient on the market causes nausea. Rather than deliberately giving people something that might make them nauseated, we would find those who already take the ingredient in their everyday lives. This group is called the cohort. We would also need a comparison group of people who do not have exposure to the ingredient. And we would then compare statistics. If the rate of nausea is higher in the herbal cohort, it suggests an association between the herbal supplement and nausea. Epidemiological studies are great tools to study the health effects of almost anything, without directly interfering in people’s lives or assigning them to potentially dangerous exposures. So, why can’t we rely on these studies to establish causal relationships between substances and their effects on health? The problem is that even the best conducted epidemiological studies have inherent flaws. Precisely because the test subjects are not randomly assigned to their groups. For example, if the cohort in our herbal study consisted of people who took the supplement for health reasons, they may have already had higher rates of nausea than the other people in the sample. Or the cohort group could’ve been composed of people who shop at health food stores and have different diets or better access to healthcare. These factors that can affect results, in addition to the factor being studied, are known as confounding variables. These two major pitfalls, combined with more general dangers, such as conflicts of interest or selective use of data, can make the findings of any particular epidemiological study suspect, and a good study must go out of its way to prove that its authors have taken steps to eliminate these types of errors. But even when this has been done, the very nature of epidemiological studies, which examine differences between preexisting groups, rather than deliberately inducing changes within the same individuals, means that a single study can only demonstrate a correlation between a substance and a health outcome, rather than a true cause and effect relationship. At the end of the day, epidemiological studies have served as excellent guides to public health, alerting us to critical health hazards, such as smoking, asbestos, lead, and many more. But these were demonstrated through multiple, well-conducted epidemiological studies, all pointing in the same direction. So, the next time you see a headline about a new miracle cure or the terrible danger posed by an everyday substance, try to learn more about the original study and the limitations inherent in any epidemiological study or clinical trial before jumping to conclusions.

96 comments

  1. 0:53 "[…] the subjects are randomly assigned to their study groups. They are often blinded […]" >:O

    😉

  2. Thank you for this video, I am tired of trying to explain this to people and they still do not get it

  3. It's especially annoying when people try to dismiss hundreds of studies of various types that came to the same conclusion because of their belief that confounding variables, supposed conflict of interest, and selection bias make them all suspect, but then those people cling to the one or two studies that share their pre-disposed bias, even though those studies have an extremely clear list of things wrong with them, far more clear issues than the hundreds of studies that said the opposite. 

  4. And now imagine fields of research like social science, how much worse the things there are, where it sometimes seems to me that every study I read is just crap and the authors just take the data they like to prove their points on gender studies or things like that…

  5. And that's why I always ignore findings of epidemiological studies. How they get away with calling them scientific is beyond my comprehension.

  6. problem is, once you research the study you'll find it's all about profit. nobody ever invest money into something that is guaranteed to have 0 return. .. there will never be a cure for cancer and you can take that to the bank.

  7. yeah if you read short resumed version of original study you might understand it wrong and make wrong conclusions

  8. Interesting and it will help me be better educated on studies as the come down that pike! This is why I subscribe to this channel!!!

  9. Last scene is not statistically significant – instead of happy face he should have dropped book into litter bin to show more natural reaction for most actual study results.

  10. This is a good video. It really bugs me when people start a sentence with "studies show blah blah blah blah" as if the words "studies show" make what they say any more credible without more information on these "studies".

  11. awesome talk! it always slightly annoyed me hearing news reporters quoting scientific studies as its d word of "god". They usually do it to  get more views on youtube or higher ratings on network tv. Truth be told there are alot of garbarge studies out there, & most ppl dont have d technical expertise to know good science from bad science!

  12. Randomized Clinical Trials do not show cause and effect! 
    All studies no matter their type can do no better than a correlation. Albeit that correlation can have varying strength and a randomized clinical trial does show the case where quite a few variables were eliminated. 

    Unfortunately, one can not eliminate all variables since you likely did not include someone from each and every culture and ethnicity on the globe. It's possible the drug may have the opposite effect in a culture with completely different diets and backgrounds. 

    There are always unknowns and a randomized clinical trial does not get rid of all of them but it is the best we can get.

  13. What about sex? Isn't it silly to lump male and female together when testing drugs? Example: Ambien and other sleep meds.

  14. It's great that TED has addressed this issue. Now people can stop pontificating their arguments with ambiguous studies. 

  15. Thank you SOOOO Much for creating this video!!!  I wish everyone thought about this before making hasty conclusions!

  16. How is it that there is no subtitles in this?! I would like to share this video with non-english speaking people. How can I help translate this?

  17. RCT is more commonly known as randomized controlled trial rather than randomized clinical trial. 

    Although the "RCT" name is sometimes expanded as "randomized clinical trial" or "randomized comparative trial", the methodologically sound practice, to avoid ambiguity in the scientific literature, is to retain "control" in the definition of "RCT" and thus reserve that name only for trials that contain controls. Not all randomized clinical trials are randomized controlled trials (and some of them could never be, in cases where controls would be impractical or unethical to institute).

  18. Proof of Global Warming scam in action, in a single glance:
    http://s6.postimg.org/jb6qe15rl/Marcott_2013_Eye_Candy.jpg

  19. This video briefly explains two kinds of scientific study: randomized clinical trials and epidemiological studies. Explaining the limitations of epidemiological studies, it encourages appropriate scepticism: if you see a headline claiming "miracle cure" or "terrible danger", try to learn more about the original study and its limitations before jumping to conclusions.

    For way more on how to evaluate studies, Nature published a list to help non-scientists "interrogate advisers and to grasp the limitations of evidence". We need more of this kind of awareness in the general population, as well as "civil servants, politicians, policy advisers and journalists":
    http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183

  20. This is a very limited skepticism though. Philosophical skepticism extends much further than looking at science, and can question the very legitimacy of science in the first place, or of our senses etc. 

  21. This is why studies did at one time show that marijuana use effected motivation. It was honestly just a bad study… Studies since have shown no such thing.

  22. THANKYOU! Now I can show my friends who shun science why their excuses of "not all scientists agree" or "science doesn't give all the answers"  or "It worked for this person, it must work for me too, so I'm gonna buy myself a $4000 water filter" why their arguments are invalid.

  23. The two phrases I remember are "correlation does not imply causation" and "take everything with a grain of salt".

    Since I have RA (I'm only 17), I am constantly bombarded with magazines and articles and random people off of the street telling me that they've found the cure because their mother's cousin's neighbor did this and now they're in remission. They'll also "tell me" that the reason I have arthritis is because I did this or because I didn't do enough of that. You have to take a step back and really do your research on new medications or diets etc. because why waste your time, energy, money, and even sometimes your health on fads that will become yesterday's news

  24. Economist Richard Wolff points out that big pharma spends many times more on sales than it does on research or production of drugs.

    The people in American drug studies are paid, thus they tend to lie about symptoms to get on a study.
     American pharmacological research is considered next to garbage by most of the world.
    In Reality, the FDA rubber stamps medical experiments on the population, or hides harm for bribes by faceless private sales psychopath bureaucrats (capitalists making mega bonuses).
     Mike Papantonio, President of the National Trial Lawyers Association, has catalogues of cases in which American science has been systematically undermined by libertarian right wing sales psychopaths.
     
    Economist J. Stiglitz points out that it is precisely this kind of harm that comes from market fundamentalism.
     information disparity increases, until it becomes [even self] deceptive rants and ads that harm everybody for profits.
     Capitalists find it far more profitable to scam, to keep people misinformed and on low wage, so they can keep them in debt and scam them even more.

  25. BLINDED? The way you represent blinded visually without explaining it is a major flaw. As a science educator, I would hope that you could fix this error and repost. Many who are ELL may take this literally and have a negative view of such studies.

  26. Please don't upvote a commenter when he or she disables replies. Singling out any topic (eg climate change) because there are limits to the usefulness of studies is stupid. All studies have to be seen in the wider context. All studies need time to be properly examined, tested and compared to other studies.

  27. Interesting video.

    I have seen firsthand how "studies" can be deliberately skewed to project a group's overall bias (ahem.. john's hopkins gun-violence and children "study"). Such ridiculous things are done to achieve their intended "facts" that I could not believe it … well, actually I could. The bogus researchers, which are often just paid shills, will only utilize precise subsets of the data and ignore large areas that would go against their personal beliefs. Other times they would outright lie, effectively re-label datasets to pad the results (counting police officer justifiable shootings as "gun violence"; counting air-rifles, BB guns and any other spring-operated toy injuries as "firearm" injuries), they also double count data or conduct telephone polls in geographic areas they know represent the same political views as them (ahem, MDA, MAIG – Michael Bloomberg organizations).

    It really is appalling when people claiming to be acredited researchers present a bogus "study" to the public, knowing full well that they cherry-picked the data to fit their pre-defined outcome. What's worse is that mainstream media jumps on the bandwagon of fear mongering, without doing any fact checking themselves.

    Looking into that one mentioned study for myself one day (about 4 hours of researching their fineprint) has really opened my eyes. Don't believe all you are told just because it claims to be a properly conducted study. Only trust something if you can verify it yourself and definitely do not trust "studies" that have tiny print at the bottom listing codes that have to be externally referenced from a separate source that you have to find!

  28. Interesting video.

    I have seen firsthand how "studies" can be deliberately skewed to project a group's overall bias (ahem.. john's hopkins gun-violence and children "study"). Such ridiculous things are done to achieve their intended "facts" that I could not believe it … well, actually I could. The bogus researchers, which are often just paid shills, will only utilize precise subsets of the data and ignore large areas that would go against their personal beliefs. Other times they would outright lie, effectively re-label datasets to pad the results (counting police officer justifiable shootings as "gun violence"; counting air-rifles, BB guns and any other spring-operated toy injuries as "firearm" injuries), they also double count data or conduct telephone polls in geographic areas they know represent the same political views as them (ahem, MDA, MAIG – Michael Bloomberg organizations).

    It really is appalling when people claiming to be acredited researchers present a bogus "study" to the public, knowing full well that they cherry-picked the data to fit their pre-defined outcome. What's worse is that mainstream media jumps on the bandwagon of fear mongering, without doing any fact checking themselves.

    Looking into that one mentioned study for myself one day (about 4 hours of researching their fineprint) has really opened my eyes. Don't believe all you are told just because it claims to be a properly conducted study. Only trust something if you can verify it yourself and definitely do not trust "studies" that have tiny print at the bottom listing codes that have to be externally referenced from a separate source that you have to find!

  29. 1:46 "People going about they usual behavior (black guy drinking grape soda)" I see what u did there! hahaha

  30. you should do a vid on the history of cigarettes, from when/why they were first made threw the crazy revenues they provided to their gradual decline and what kind of studies were used to discover their health risks!

  31. anybody else notice the black guy downing the GRAPE SODA @1:51 after they said "observed going about their usual day" lol #Racist smh

  32. its pretty hard for people with no university-level understanding of biochemistry to look up a study and form their own thoughts on the study. As a student in medicine, i have no problems (except for having to look up a term i dont get in english every now and again) with this, and i know the sites to check (but i can only check them with my university-provided-account). My friend who hardly did high school chemistry, but who is equally skeptical about studies, has no chance whatsoever to do this. He's more than bright enough, but if you look at how most of these things are written you'd think its a WW2 code language.
    Kind of sad considering most of the conclusions in studies shouldnt be hard to (re)write in human language. Am i the only one who thinks it's rather silly that in this day and age of information-abundance;  its mostly the choices you made at 16-17 that determine whether or not you can understand whats going on in science? Especially when "whats going on in science" often translates to "am i being healthy or am i doing the exact opposite".

  33. It's so sad that humankind as a whole is so constrained by social and societal factors that we are unable to find Truth efficiently

  34. I'm waiting to hear more about the study that showed that blood from young mice reversed aging in old mice, rejuvenating their muscles and brains. Are there human trials on this yet?

  35. So they don't want to bring things into the market before they have been tested for possibly harmful side-effects.

    Well, I'm glad someones looking after us.

    Time for a cigarette break

  36. Unfortunately, looking into studies is a lot easier said than done. I've tried reading actual studies to see how they came to their results, and most I came across are published in scientific journals that require you to pay money to see the contents. Most allow you to see the abstract, but not the actual contents of how the study was conducted. Additionally, even if the studies were publicly available, the studies are structured to support the conclusion in the abstract; such as only using selective data, as mentioned in the video. This makes it very difficult to discern the validity of the test because you can only see the test in the framework that the author intended.

  37. One other problem is that sometimes scientists are paid(by government or other entity) to skew a study somehow that benefits the bribing party.

  38. Remember the fat scare? Every product was made light, reducing fat but increasing sugar instead. That was a move by the sugar industry to sell more sugar. You have them to blame for diabetes. Think critically people.

  39. I'm still a bit confused with the difference between 'correlation' and 'cause-effect' stated at the almost end of this video ._.
    Anyone can explain further? Thx

  40. A brilliant video. Many of us need to learn the importance of questioning each thing that comes our way instead of falling for something blindly. This is such an easy way of breaking down the belief that every heading or an advertisement saying ' Scientifically Proven' is the best thing available on the market. Great work!

  41. As a neuropharmacologist and a professor, this is what i always teach my students at the school of medicine, but every person should also understand that. it can save lives

  42. Great video… I just want to point out that these tricks to identify clickbait titles are useful when you have a lot of time to spend for that particular study, because you have to find and read the original article, and probably you won’t understand much of it if you aren’t a doctor, so you need to find other sources or contact the authors of the article. All I want to say is… I understand that critical thinking is important, but can’t we have journalists to do this? Because I thought that was their job.

Leave a Reply

Your email address will not be published. Required fields are marked *