Studies Say…

I love this quotation( attributed to one Andrew Lang, who was born in 1844): “He uses statistics as a drunken man uses lamp-posts… for support rather than illumination.”

Actually, we all do that from time to time, and political psychologists tell us it is the mark of “confirmation bias”–the very human habit of cherry-picking available information in order to select that which confirms our preferred worldviews.

Because that is such a common behavior, and because we can easily find ourselves citing to “authorities” that are less than authoritative (and sometimes totally bogus), I’m going to bore you today by sharing information from a very useful tutorial on assessing the credibility of “studies,” as in “studies confirm that..” or “recent studies tell us that…”

Academics who have conducted peer reviews of journal submissions are well aware that many studies are fatally flawed, and should not be used as evidence for an argument or as confirmation of a theory. (If I were doing research on voter attitudes, and drew my sample–the population that I surveyed–from readers of this blog, my results would be worthless. While that might be an extreme case, many efforts at research fail because the methodology is inappropriate, the sample size is too small, the questions are posed in a confusing manner, etc.)

The tutorial suggests that journalists intending to cite to a study ask several pertinent questions before making a decision whether to rely upon the research:

The first question is whether the study has been peer-reviewed; in other words, has a group of scholars familiar with the field approved the methodology? This is not foolproof–professors can be wrong–but peer review is blind (the reviewers don’t know who conducted the study, and the authors don’t know who is reviewing it), and tends to be a good measure of reliability. If the study has been published by a well-regarded academic journal, it’s safe to assume that its conclusions are well-founded.

Other important inquiries included looking to see who funded the research in question.

 It’s important to know who sponsored the research and what role, if any, a sponsor played in the design of the study and its implementation or in decisions about how findings would be presented to the public. Authors of studies published in academic journals are required to disclose funding sources. Studies funded by organizations such as the National Science Foundation tend to be trustworthy because the funding process itself is subject to an exhaustive peer-review process.

The source of funding is especially relevant to the possibility that the authors have a conflict of interest. (Remember those “studies” exonerating tobacco from causing cancer? Surprise! They were paid for by the tobacco companies.)

Other important elements in the evaluation may include the age of the study, since, as the post noted,  “In certain fields — for example, chemistry or public opinion — a study that is several years old may no longer be reliable.”

Sample size and the method used to select survey respondents are obviously important, and statistical conclusions should be presented in a way that allows readers to review their calculations. It’s also worth looking closely to see whether the study’s conclusions are actually supported by the reported data. As the post notes,

Good researchers are very cautious in describing their conclusions – because they want to convey exactly what they learned. Sometimes, however, researchers might exaggerate or minimize their findings or there will be a discrepancy between what an author claims to have found and what the data suggests.

In an information environment increasingly characterized by misleading claims, spin and outright propaganda, the ability to distinguish trustworthy research findings from those that are intellectually suspect or dishonest is fast becoming an essential skill.

Comments

Who Do You Trust?

In its business section this morning, the New York Times had a lengthy story about Angie’s List, the Indianapolis-based company that offers members access to reviews of service providers of various kinds. The reviews are provided by the members, and the article noted that–unlike sites like Yelp!–those reviews are not anonymous. While Angie’s list doesn’t publish the names of the reviewers, it does insist that evaluations come from identifiable individuals.

As the company’s public offering explained, they require this because it’s hard to trust anonymous statements and “facts” culled from the internet. The insistence that reviews come from verifiable sources is one way to increase the trustworthiness of the information being provided.

Lack of trust may be the signal characteristic of our times.

Angie’s List isn’t the only organization trying to deal with the wild west that is our current information landscape–far from it.  I would argue that much of what ails America these days is either enabled by or a direct result of mis-information, dis-information and  information uncertainty.

We are all being constantly bombarded with “news” that we aren’t quite sure we can trust.

The days when Mr. and Mrs. America tuned in to Walter Cronkite–and relied on the accuracy of his reporting–are long gone. Newspapers–with a few exceptions–fill the few pages they still publish with restaurant reviews and diet tips rather than fact-checked reporting. Cable “news” is anything but; it’s spin and talking points, and most Americans recognize that. It is increasingly difficult to determine the credibility of information we find online. No matter how goofy the perspectives or bizarre the conspiracy theories, you are likely to find confirmation of them in some fevered corner of the internet. (Just ask Rep. Bob Morris, who found “evidence” confirming his suspicions about those sneaky, abortion-loving, lesbian Girl Scouts.)

The problem is, when we no longer have authoritative sources and institutions we trust, societies don’t work very well. We lose an essential element of what social scientists call “social capital.” (Warning: shameless plug approaching.) I wrote about the causes and troubling consequences of diminished social and institutional trust a couple of years ago, in my book Distrust, American Style.

It isn’t just the folks who find internet confirmation that aliens landed in Roswell and the government covered it up. We’ve always had conspiracy theorists who are, shall we say, lightly tethered to reality. Today’s information landscape promises consequences far more pernicious than enabling the Holocaust deniers or encouraging the religious zealots convinced that the Rapture is imminent.

When ordinary Americans can’t be sure who is telling the truth, it’s easier to retreat into “us versus them” views of the world; easier to believe that a President who doesn’t look like you is really a secret Kenyan Muslim; easier to believe that an effort to provide healthcare is really an attack on religious liberty.

There is broad recognition that we have a problem.

The question is: how do we fix it?

Comments

Interesting Observation

My granddaughter Sarah currently lives in Wales; she is attending the University of Wales and will graduate this summer. She reads the Guardian, and this morning sent me the following text message.

“Taken from the comment and debate section of the Guardian this morning: ‘what you need to say and do to be credible in the Republican Party essentially deprives you of credibility outside it. The Republicans recognize this, but like an obese glutton at an all-you-can-eat buffet, they just can’t seem to help themselves.'”

The comment was in response to an article on “The American Right, Stuck in a Hyperbolic World,” and I think it captured the current dynamic perfectly. Right now, for example, it looks quite likely that the House GOP will shut down government, despite Democrats’ willingness to meet their demands halfway. (The Republicans want 60 billion in cuts; Democrats are offering 30 billion.) They seem absolutely oblivious to the damage indiscriminate cuts will do to the still-fragile recovery–and equally oblivious to the political damage their posturing is inflicting.

As the commenter noted, they just can’t help themselves.

Comments