An online publication called “Journalists’ Resource” recently posted a really helpful article about academic studies.
By this time, most sentient beings understand that we live in a world ruled by confirmation bias: the process of cherry-picking data in order to confirm our pre-existing beliefs and prejudices. We all do it–us “good guys” as well as those who (since they disagree with us) are clearly wrong. And when we encounter an academic study that confirms our positions, we’re excited.
See? I was right!
For those of us who do try to seek out different perspectives, who make an effort to step back and be analytical and measured, credible research is important. The problem is, not all research is reliable, and relatively few people have the statistical and methodological skills to assess the credibility of a given study.
That’s why this article is so useful. It gives journalists–and by extension, the rest of us–a “map” for determining whether and to what extent a study’s conclusions are reliable.
This requires data literacy, some familiarity with statistical terms and a basic knowledge of hypothesis testing and construction of theories.
Journalists should also be well aware that most academic research contains careful qualifications about findings. The common complaint from scientists and social scientists is that news media tend to pump up findings and hype studies through catchy headlines, distorting public understanding. But landmark studies sometimes do no more than tighten the margin of error around a given measurement — not inherently flashy, but intriguing to an audience if explained with rich context and clear presentation.
The rest of the article is well worth reading; it lists the questions one should ask, defines scholarly terms, and provides context for figuring out what a particular study is really telling us. Or not.
Really helpful–assuming we are looking for information, and not just ammunition.
Good Morning, Sheila………I first considered letting you immediately know how I view MY study to all the rest……but then I said to MYSELF….”I have all eternity to do that so why interrupt her morning coffee routine?”
Nice article. Quantitative methods in a box. People need to read this, put a copy to the side, and then actually read some of those studies. They don’t have to read an entire 30 page study, but if they follow the article, they will be fine.
Well-done, peer reviewed studies don’t conflict very much, and if they do, the authors try to address the differences, because they have peers who are trying to understand the area, too. While ideological issues are not unknown, areas where there is serious study going on are not generally adversarial shoot-outs. That is one reason why people need to pay attention read the climate change articles. For scientists, climate change study is not a shoot out.
If you just scan the International Panel on Climate Change (IPCC) report, you will see a broad number of areas studied by a huge number of people who are not into shoot outs. Many of the studies cited are meta-analyses–attempts to pull together a large number of studies. The report is technical, but they want people to understand it, so they do all kinds of things to make it readable. Articles tend to add on to each area, and climate scientists tend to work together on issues, as in most areas of serious study. Being scientifically literate doesn’t happen overnight, but if you work at it, you will understand more.
Bah.
All the best disciplines are a priori.
I believe that there are many contributors to our contemporary feeble mindedness. Confirmation bias is certainly one.
Research of the thoughts of others has always consisted of finding then evaluating sources. Google and the Internet have greatly simplified the finding part. The easy part. What used to take weeks can be done in minutes. Library Science has been automated but processing the contents of the library requires so much more.
For one thing there is so much more to process. Authors used to be just a few of us now they’re all of us. Anybody can now launch a thought virus for the world to ponder.
Then there is the personal issue of who to grant credibility to one aspect of which is confirmation bias. Which is made worse by the fact that our most helpful librarian robot tends to offer us more from those who agree with us than those who disagree. He knows us too well.
Then there is our miserably incomplete education in statistics. Most people without statistical education are putty in the hands of those with even a moderate amount. They can be led to believe almost anything.
How about the explosion of automation of graphics? Graphs and pictures and maps and illustrations. Want a professional looking presentation? Push a button.
All of this occurs in a soup of media flashing in front of our ears and eyes most every minute of every day. Information not requested but pushed into our subconsciousness by people so skilled at that that we will deny being influenced even while our brain is being rearranged to suit their, not our, fancy.
We have lost our minds. Literally.
It’s probably not too far from the truth that in the near future a PhD will be the new high school diploma. Unfortunately that means even more specialists with deeper understanding of narrower fields. And therefore more need for each other to comprehend the big picture.
To say that we are ill equipped for the future is understatement. That’s why so many are unable to leave the past. Unfortunately the stakes we face have never been higher. Survival has overtaken comfort.
Can those who have been taught research skills lead those who haven’t developed them away from their shiny objects back to the truth, whole and nothing but?
Good question to research.
If one wants to know whether a study is biased, look at who funded the study. If you read a study commissioned by the National Institute on Drug Abuse (NIDA) on Marijuana, you can be pretty sure it will be negative. They’ve done that since their inception, and are usually very flawed.
Good point, Neal.
The only reason anyone undertakes a study is because they’re interested in the subject.
I think I can say with a 99% Level of Confidence the stenographers (called Reporters) at The Star and Readers on our TV Media (called Reporters) will not challenge the dubious experts that always conclude the latest Sports Venue that needs to be built will generate millions in revenue. We have now a days have the Rent an Expert, which will draw the conclusion that is desired. (Read Tobacco Scientists – the type of “Scientists” that tried to cover-up the links between smoking and lung cancer.
If you have been around Indianapolis long enough you will recall all the hype and “expert” studies that always conclude that building a New Sports Stadium will be a huge benefit to Indianapolis. Neither The Star or our TV Media will ever present or list all the Corporate Welfare in it’s various forms that has been and still is lavished on down town.
There’s a reason for our collective feeblemindedness. We get much more exposure to advertising than education.
https://youtu.be/bEULSwE5GEk
As good a presentation of the source of our weakness as I’ve seen.
Someone I know worked for a private firm and did a study for a client who hoped the study would facilitate development of a project that could impact water quality. When the study results did not support the project’s objective, the boss ordered the findings altered. He stated, “the client doesn’t pay for negative results.” Ever since then, I have viewed any study with a jaundiced eye. Who ordered it and who paid for it? Answers to those two questions should be in the first paragraph of any study and any reporting on said study.
USA journalists cannot put together any body of literature for having a School of Journal(ism) that attracts enough professional philosophers and doctors of journalism to award doctoral degrees in the College disciplines. When deans try to sort their plans, they cannot tell much difference from language(s) for which there are plenty of teachers, lexicographers, any kind of -graph- that “we” can find clearly posted, visible, tangible or can be rendered (graphed) tangible. In other worlds, they are employed to teach readers English, French, Spanish, (Roman) “Chinese” and the American English for (pictographs). No electricity, no philosophy, history, quotations. . . We do not bother to even check periodicals and journals for perennial responsibilities to study, such as humane legal homicide. . .
The phrase “research has shown” often means that “I read an article in an advocacy journal in which the writer’s opinion was…” No question that you need to look at who funded the paper, but peer reviewed journals require the writer to reveal that information. Like anything else, you read all sources carefully in an intelligent way. It’s important to screen the stuff you are going to cite, and be critical about it before you plunge in and swallow it whole.
On the other hand, before dismissing all research studies, and throwing out the baby with the bathwater, it’s best to actually read some of them instead of remaining ignorant.
When you hear someone who claims that “the research has shown”, simply ask them which research they have read on the subject, and if they have read anything at all besides someone’s opinion piece.
As a former academic scientist, I can say that the only people that we despised more than journalists was our own university PR department. Many of the exaggerations, mis-statements, and not-even-wrong nonsense was written first by some idiot across campus who had no idea what we did & hadn’t even bothered to google it. In two cases I’m familiar with, press releases went out, presumably to “make us look good,” without a single email, phone call, or contact of any kind.