I love this quotation( attributed to one Andrew Lang, who was born in 1844): “He uses statistics as a drunken man uses lamp-posts… for support rather than illumination.”
Actually, we all do that from time to time, and political psychologists tell us it is the mark of “confirmation bias”–the very human habit of cherry-picking available information in order to select that which confirms our preferred worldviews.
Because that is such a common behavior, and because we can easily find ourselves citing to “authorities” that are less than authoritative (and sometimes totally bogus), I’m going to bore you today by sharing information from a very useful tutorial on assessing the credibility of “studies,” as in “studies confirm that..” or “recent studies tell us that…”
Academics who have conducted peer reviews of journal submissions are well aware that many studies are fatally flawed, and should not be used as evidence for an argument or as confirmation of a theory. (If I were doing research on voter attitudes, and drew my sample–the population that I surveyed–from readers of this blog, my results would be worthless. While that might be an extreme case, many efforts at research fail because the methodology is inappropriate, the sample size is too small, the questions are posed in a confusing manner, etc.)
The tutorial suggests that journalists intending to cite to a study ask several pertinent questions before making a decision whether to rely upon the research:
The first question is whether the study has been peer-reviewed; in other words, has a group of scholars familiar with the field approved the methodology? This is not foolproof–professors can be wrong–but peer review is blind (the reviewers don’t know who conducted the study, and the authors don’t know who is reviewing it), and tends to be a good measure of reliability. If the study has been published by a well-regarded academic journal, it’s safe to assume that its conclusions are well-founded.
Other important inquiries included looking to see who funded the research in question.
It’s important to know who sponsored the research and what role, if any, a sponsor played in the design of the study and its implementation or in decisions about how findings would be presented to the public. Authors of studies published in academic journals are required to disclose funding sources. Studies funded by organizations such as the National Science Foundation tend to be trustworthy because the funding process itself is subject to an exhaustive peer-review process.
The source of funding is especially relevant to the possibility that the authors have a conflict of interest. (Remember those “studies” exonerating tobacco from causing cancer? Surprise! They were paid for by the tobacco companies.)
Other important elements in the evaluation may include the age of the study, since, as the post noted, “In certain fields — for example, chemistry or public opinion — a study that is several years old may no longer be reliable.”
Sample size and the method used to select survey respondents are obviously important, and statistical conclusions should be presented in a way that allows readers to review their calculations. It’s also worth looking closely to see whether the study’s conclusions are actually supported by the reported data. As the post notes,
Good researchers are very cautious in describing their conclusions – because they want to convey exactly what they learned. Sometimes, however, researchers might exaggerate or minimize their findings or there will be a discrepancy between what an author claims to have found and what the data suggests.
In an information environment increasingly characterized by misleading claims, spin and outright propaganda, the ability to distinguish trustworthy research findings from those that are intellectually suspect or dishonest is fast becoming an essential skill.
Sheila: “In an information environment increasingly characterized by misleading claims, spin and outright propaganda, the ability to distinguish trustworthy research findings from those that are intellectually suspect or dishonest is fast becoming an essential skill.”
I agree. But, I question the ability of academia to make that decision. Although it was many years ago, my partner Professor Harvey Wheeler stood up for his principles while a professor at Harvard, was dismissed and replaced by Henry Kissinger. So much for peer review in that situation.
A bigger problem would be to distinguish trustworthy research findings that are TRUE and all the implications that would follow. Universities, including Harvard, are not exempt from bias, especially political bias.
In my estimation, the only answer is to have IRREFUTABLE proof of your findings, which in most cases, BUT NOT ALL, might be impossible.
It is often difficult to find out who funded a study. In 40 years of Cannabis research, when we can finally learn who funded anti-Cannabis studies, it’s almost always opponents, funded by big pharma or the National Institute on Drug Abuse (NIDA) whose goal is to prevent relegalization. There are over 3,000 studies showing the safety and efficacy of Cannabis as a medicine, but the vast majority of those studies are done in other countries, notably Israel and Spain, and are not officially accepted by the U.S. government.
The most recent American study claiming to show Cannabis negatively impacts IQ had 20 some respondents. No truth to be found there.
For all issues, this is a major problem.
Mea culpa; I will believe every word from the mouth of Bernie Sanders before considering any of the spinning views from the pursed lips – or Tweeting fingers – of Trump.’
A wise friend told me long ago that I can find statistics to prove any point I choose if I know where to look. I am amazed at the number of polls I see resulting in support for Trump’s questionable, constantly changing views, actions and inactions. Who conducts these polls, who selects those conducting the polls, who do they poll and who selects those to be polled? The polls I have responded to on line and by mail never appear in any poll results made public on vital issues today. And, I only respond to qualified, reputable organization’s surveys.
Sheila: “In an information environment increasingly characterized by misleading claims, spin and outright propaganda, the ability to distinguish trustworthy research findings from those that are intellectually suspect or dishonest is fast becoming an essential skill.”
My only argument with Sheila’s comment is; this is not an ability that is inborn or a skill which can be learned. The depth of complete research to reach the truth between propaganda and facts would prevent finding answers in a timely manner, especially today.
Marv: “In my estimation, the only answer is to have IRREFUTABLE proof of your findings, which in most cases, BUT NOT ALL, might be impossible.”
My only argument with Marv’s statement is his use of the word “might”; irrefutable proof of any findings would require polling everyone and every source of information on any issue and, again, could not be done in a timely manner to render results when needed.
A post on Facebook this morning regarding a BBC interview with a Mr. Ford (sorry, I have forgotten his title) who refuted the reason for Trump’s bombing in Syria being due to the accusation that Assad had used chemical weapons on his own countrymen – and women and children. Could have been their own bombs accidentally hitting the chemical warfare storage areas was one possibility he offered. Where he got his information, if he had information, was not revealed. Only his comments refuting or questioning everything we have been reading and watching on national and international news; but it gives us some new points to ponder. Time for another poll?
I have found it to be extremely difficult, if not impossible, to find studies or research that proves to be fair and impartial. Can anyone on this blog actually point out one that was not funded by corporations wanting to gain from a study and having paid the researchers? University personnel conducting “research studies” are frequently proven to have conducted their research at the behest of corporate interests who paid them for their efforts.
JoAnn @8:06 a.m.,
Could you further explain: “My only argument with Marv’s statement is his use of the word “might”; irrefutable proof of any findings would require polling everyone and every source of information on any issue and, again, could not be done in a timely manner to render results when needed.”
It appears to me “My only argument” is an attempt to disprove everything I had just said. Nothing new, you’ve been doing that for almost two years.
The gop stopped sending questionaires to me over a year ago after I repeatedly responded with anger at their leading questions. Basically, I was ripping them a new one on each questionaire. They eventually got the message not to send them to me.
Their polls typically ask questions and provide answer “choices” that lead their constituents to provide the answers that they want. They can then cite these polls as reflecting what the public wants. It is all BS.
My older daughter works for an agency in Washington and is immersed in statistics. Once I asked her what would happen to our social security system if on average everybody lived to be as old as Methuselah, who lived to be 969 years old per the bible. She answered as any good statistician would: “No problem, Dad. You just work until you are 950.” I promptly replied that I wasn’t working nine centuries plus for a lousy social security check! This exchange points up the excellent quote Sheila has made today as innovative but (as Marv rightly suggests) impossible in application to all matters the subject of research. My wife’s doctoral thesis on the amalgamation of reading and content did not yield the results we had hoped for, but she dutifully reported such results as doubtful proof of her hypothesis. Sample size, inherent bias and old and currently inapplicable studies studies are just some of the tools and results human researchers employ. Republicans, for instance, are wont to use average wages rather than median wages as a defense to calls for minimum wages. Right. Warrent Buffet However, I would not have it any other way because we should not sacrifice the good for the perfect (and who defines even perfect). Parenthetically, I am preparing to attend the Democratic Women’s tax march here in southwest Florida this morning. How about you, my fellow commenators?
JoAnn:”My only argument with Sheila’s comment is; this is not an ability that is inborn or a skill which can be learned.” By that logic we should just give up now and not even try to see past the end of our own noses. The point is there are better studies, good studes, poor studies, and of course heretical ones. There’s really no possibility of a perfect study of any subject for a long list of reasons. But better IS possible and your quality of life depends on it. So we commit to doing the best we can given the resources at hand. It’s a tough slog to maintain, but I’ll never accept that research or learning how to do it better is not a worthy endeavor or impossible. Note: Spoken as a NIH funded biochemist. Well, as.ling as the NIH is still a thing you know.
mea culpa – I hit the wrong button and prematurely published with Warren Buffet. What I meant to write and didn’t was that when the compenation of Warren Buffet, Bill Gates and other such superrich people are averaged into the pot with the rest of us, the skewed result gives an unfair portrayal of those who are suffering from wage inequality. Also, there is a typo on the last word – it should read commentators. My defense is that it is early and that I am thinking of the march.
Gerald,
Why is it that I haven’t been able to successfully click your website? Is it the period before Gerald [.G]?
Poll results are generally untrustworthy due to their very nature. Scientific research in math, chemistry, biology, medicine, etc. is another matter altogether. Reputable journals are seen as reputable by everyone in the field. Still, we occasionally get bad science, even in the best of the journals. That is due to unscrupulous or desparate scientists using bad data. Just ask “The Lancet” about Dr. Wakefield. That’s why the last and most critical question is, “Is it replicable?” That question is generally answered post publication and sometimes leads to refutation of an earlier article.
Take some good advice from my dad, “Believe none of what you hear and only half of what you see.”
JM; “There’s really no possibility of a perfect study of any subject for a long list of reasons.” Your comment says what I was trying to get across and is more to the point. Thank you.
Marv; you should know me well enough to know if I meant to attempt to “disprove” everything you said I would state it outright. It is impossible to disprove everything you said because you are an intelligent man with high qualifications and years of experience in legal matters and I have NOT been trying to disprove your views for two years. You do not like or approve of my support for Southern Poverty Law Center because they do not do what you believe should be their primary goal – fighting anti-Semitism. It IS one of their many issues and is one I fully support; I do not support every issue they work on. I do not agree with everything stated by everyone on this blog, including you and sometimes Sheila’s views, no one else has turned on me for disagreeing – they simply state their disagreements. This is called freedom of speech; you are perfectly free to continue accusing me of whatever comes to your mind. You have several times referred to the “veiled threats” against you by author John Grisham; I recognize your “veiled disagreements” with my statements but accept them as your personal views and make no accusations against you and will not. It is because I disagreed with one word, “might”, in your statement that I copied and pasted your full comment to keep my remark in context.
To Sheila and all others on the blog; I apologize for making this personal conversation public but there is no other way to respond to Marv. I hope this ends this on-going difference of agreement regarding SPLC between he and I. Another mea culpa?
In my original post at 7:28 a.m. I wasn’t referring to polling. I never mentioned the words poll or polling.
You have to give JoAnn all the credit she deserves. She’s one clever woman.
We have lots of clever people who comment on this blog. Edifying.
While this may be ancillary to the comments on Sheila’s blog post a all of this reminds me of all the studies that I read working at Riley Hospital for Children regarding the prevalence of autism and autism spectrum disorders in the general population and the causes for autism. The studies were the result of research and various medical schools and hospitals across the country and were generally all over the map in terms of their conclusions. It complicated things in regard to properly funding and orchestrating efforts to work with children on the spectrum it was a source of great confusion for their parents since they really wanted, most obviously, to know what was going on. No doubt the researchers of what proved to be highly erroneous studies were all well paid and tenured they caused utter chaos for those who are charged with dealing with this medical issue.
All the way through all of it money was a big factor both in getting Federal funding for research and also for Big Pharma come up with all the various medications that was supposed to deal with the various aspects of the autism spectrum. They all made out like bandits but ultimately a lot of their results were ultimately refuted discredited. The community of medical professionals, family advocates, family resource managers like myself, as well as parents and patients, had to deal with all the wreckage that they placed in our way regarding effectively helping those that needed our help with us having no other motives than just that.
P.S. Mucho kudos to Jo Ann for her very sharp mind and to Marv for recognizing her for having it.
Tom; first, thank you for the complement.
In your references to autism studies, did you ever find studies regarding effects of Terbutaline on fetus causing autism-related or similar disabilities? I would be interested in this information; there seems to be little on line even though someone on the blog was familiar with a law suit 30 years ago which the family lost due to the lack of studies on the issue. My grandson, soon to be 18, has disabilities which point to Terbutaline as the cause. Doctors prescribed it for pregnant women to prevent premature contractions; it had NOT been approved by FDA for this use. Later studies did show the drug caused heart problems for the mothers and much later studies were few regarding effects on the fetus.
Freddie de Boer, that newly minted PhD who I told you about a year or two ago, is addressing exactly this epistemological question at his new ANOVA blog: fredrikdeboer.com
He’s decided to forego his radical past, mainly because of the online hate storms, and to focus instead on education & what’s evidentiary there.
see http://www.jacobinmag.com/2017/04/charter-schools-nashville-neoliberalism-education-reform-teachers-unions/
I am a proud Jacobin subscriber who’s saving up for a Baffler subscription. I am so left I walk into traffic.
Years ago a young man of my acquaintance, fresh out of college, went to work for a company of good repute. His job was to conduct studies on a site a client purposed as suitable for industrial waste disposal. After several weeks of work and analysis, he concluded that the site was not appropriate for that purpose due to the composition of the soil and the underlying aquifer and it would be too close to a school’s water supply. When his supervisor read the report, he handed it back and said that the conclusion shown was not what the client was paying for. The young man was directed to write the desired conclusion.
The fact is that studies done by private concerns with funding from a client with a specific desired outcome can be compromised. Academic studies in public institutions are subject to peer review and are less likely to pass flawed conclusions along. It may take some time for flaws to be exposed, but they almost always will be.
JD; Flint, Michigan, immediately came to mind. Along with Karen Silkwood and Kerr-McGee, Erin Brochovich and Pacific Gas and Electric, Jon Schlictmann and Beatrice foods. Closer to home the small cancer cluster in Johnson County a year or so ago which pubic health officials did not recognize as a cancer cluster due to different forms of cancer in the patients which is typical in all hazardous waste effected areas. I am but one person and my sister-in-law’s 11 year old nephew is one of the Johnson County victims still being treated for cancer; my friend’s nephew WAS another one, he died on his third birthday last summer. Evidently no study was done about hazardous materials in the groundwater in that small area of Johnson County because their decision was reached in a matter of a few days. This country is doomed due to bogus studies and lying reports such as you reported, cancers are epidemic in this country. The escalation has happened in recent decades as more and more businesses dump hazardous waste wherever it is convenient – and cheapest – for them.
“Studies Say…” otherwise as the death toll of our families rises and privately owned politicians keep big business in business and now the EPA is all but repealed. Well; we may not have to worry about it much longer; Trump is working on “nuking ’em” between his weekend golf games, just deciding on his target. What do the “Studies Say…” about surviving nuclear war?
Life is hard and complicated and gets more so everyday. Our collective body of knowledge is expanding so fast that the only way any individual can keep up is to be extremely narrow and constantly learning.
The net effect is that every one of us becomes more dependent everyday on others to know what we need to in order to navigate life’s waters.
That makes collaboration an absolutely essential life skill.
But, being the contrary beasts we are we’ve picked this time in history to go the opposite way, becoming media created ornery blind as a bat minions.
It’s crazy. Can we survive until our culture swings back to functional?
Only maybe.
Marv – the dot? Probably. I have removed it.
Thank you for this information and the link to the tutorial. It was all good until the last point regarding meta-analyses which raised red flags. If there is analysis of each study in a grouping of good studies, that would prove beneficial, but I must take issue with quantity over quality. Too often some simply will tally studies or surveys with a hodgepodge of methodology and wide varieties in samples to draw conclusions which would be very hard to justify on closer examination. Some go even further to average the results – very bad idea. Particularly when in search of solutions, averaging away all the nuances in addressing a problem can be disastrous.
Additionally, reliance on rankings sometimes can be misleading. A friendly legislator was once accused of missing half the committee meetings to which he was assigned. On investigation, I learned the committee only met twice, and this legislator missed one meeting to be with his daughter who had surgery that day. I supported the legislator who chose to be a good father.
Another example – a lobbyist cited numerous studies showing that lowering class-sizes in public schools made no difference in student achievement. But then we found none of the studies included a class-size small enough to offer individualized attention to student achievement. When Indiana actually implemented a small class-size pilot program in grades K-3 with class-sizes of 18 or fewer students and compiled the results, the students showed improvement in achievement AND attendance AND student behaviors. Teacher attendance, innovation, and morale all improved too.
Sometimes we could avoid the cost of expensive studies by just listening to practitioners.
We did not use anonyomous records of any kind at IU for “blind” work at all. I did only “blind” work for Brown Williamson Research and Development chemists and physical scientists on 300-year projects (then) of the British-American Tobacco Company FAMILY members and directors of the stock companies at New York-London offices. They are not the corporate copyrighters of packages for the factory products once taxed. As with the Watergate records, those may be private but are not anonymous. Nor are those aging 17-year-olds who did not waive their permissions to be crefdited for being ignorant of “geography” in New York terms! Or, What do 17-year-olds know AFTER their doctorates in cartography!
Gerald,
Marv – the dot? Probably. I have removed it.
I still haven’t been able to connect to your site. Did you use http://www.? Maybe that’s the problem. Are others on the blog having the same problem?
Marv – The problem may have been on my http. I think I have corrected it. Check me out.
Gerald,
I’m still having the same problem. I believe it is on your end. I tried a few other similar situations and they workd o.k. Did you try it?
Thanks, Sheila, for another excellent post. I couldn’t agree more.
However, as a retired scientist, I do want to warn against thinking that everything is bogus. (Thank you Peggy Hannon).
While there have even been a few articles retracted from the prestigious journal, Science, recently, academic science does strive to be self-correcting. So to expand upon Sheila’s point, it sometimes doesn’t take years for a study to be shown to be false.
As a counterweight, some years ago, two studies were conducted comparing the cholesterol lowering drugs Paravachol and Lipitor. Pravachol manufacturer Bristol-Myers Squibb funded one while Lipitor manufacturer Pfizer funded the other. Both were published.
Both showed that Lipitor was superior to Pravachol. Sometimes scientific integrity beats funding sources — sometimes.