I don’t recall where I saw the link to the Harper’s article, The revolution will not be pasteurized: Inside the raw-milk underground, but I’m glad it came to my attention. The article is quite long, but very interesting, and mostly excellent. Oddly enough, it isn’t the subject of raw milk that leads me to write about the article.
The last two paragraphs are what shouted to me; they’re presented in abridged form below.
But when it comes down to it, here’s why I drank the raw milk. The sun had just come up, and we’d already finished three hours of work in the barn. I was filled with a righteous hunger. .... Although dairy isn’t for everyone, I come from the people of the udder: my ancestors relied so heavily on milk that they passed down a mutation allowing me to digest lactose. For many generations my forefathers sat down to meals like this after the morning milking. It felt unambiguously right.
This, of course, is the very definition of bias: the conflation of what feels right with what is scientifically correct. But as it was, I could only hope that my biases were rooted in something more than nostalgia. .... Someday the uncertainties of dietary science will fall to manageable levels, but until then I will rely on my gut. I drained my cup and poured thick clabbered milk and apple syrup on my porridge. If any bacteria disagreed with my body, the conflict was too small to detect.
Hate to break it to you, Mr. Johnson, but that isn’t the most common definition of bias in science. Usually it refers to the researchers skewing methodology and/or reporting to favor a particular result or interpretation. But that’s the least of my gripe with your words.
What, exactly, does “scientifically correct” mean? Contrary to strong popular myth, scientific methods do not definitively prove things—they rely on probabilistically-based tests and careful experimentation to find a result, and give an idea of how likely that finding is due to chance factors. That’s what the p level indicates; and it used to be (I’ve been out of this field for a long time) that the minimum standard for significance in most studies was .05—meaning that 5 times out of 100, the results could have been obtained by chance.
There isn’t a friendly sign that indicates when chance factors rather than the experimental manipulation(s) have led to a statistically significant result, so researchers try to control the research conditions as much as possible; and more important, studies are often repeated to some degree, to help address this issue. (One problem, however, is that absolute replication would be expensive and is often seen as not worth it, so subsequent studies may address the same issue but in a slightly different way. Almost certainly the test subjects are different, for starters.) It’s really difficult to assess whether studies that touch on the same issue(s) are really close enough to count as a replication. This is just one issue I have with meta studies—research that gathers all the data from many studies examining a particular question, and looks at that pooled information to try to definitively address the research question.
Another tool in the researcher’s kit is power—setting up the study so that if meaningful effects are present, they will be found in the statistical analysis. One easy way to increase power is to increase the sample size—the number of subjects tested. Having too many participants can lead to problems, but the cost of research generally precludes that situation occurring.
The important thing to understand is that the statistical results, and the conclusions the researcher draws from them, are based on aggregate information—in most tests, the mean and standard deviation (or similar measures of central tendency and variability) are the core figures underlying the statistical analysis. In fact, a careful researcher will have established, in advance of collecting any data, a set of criteria for eliminating outliers from the analysis. In most cases that’s probably wise, but sometimes, those outliers may be where the most interesting information resides.
So, given this aggregate-based methodology, what can one study—or even a whole raft of studies on the issue—say about any given individual’s likely behavior under the studied conditions? Not much, actually. Speaking probabilistically, the way to bet is that each individual will be close to the mean, but that assumes a so-called normal distribution (that nice bell curve that so much science is based upon), and it assumes that individuals are more alike than they really are. [It is an interesting side note to wonder exactly how common a symmetrical distribution is in nature. Given that hand dominance in humans is highly skewed—decidedly not a normal distribution—one wonders why it is so universally assumed to hold for just about everything else.]
Also, science in a lab can be very far removed from the conditions of everyday life. As a brief example, in my former field of expertise, tachistoscopic presentation of letter combinations for very brief periods (we’re talking milliseconds here) has led to various theories of reading and general perceiving: but does anyone really think that the bulk of our daily reading/perceptual experience is captured by those static, barely-there stimuli?
I’m getting rather far afield from my basic point, which is this: scientific findings rely on aggregate information often gathered under artificial conditions. At best, they offer ideas about general human functioning (again, an interesting side topic for exploration, since many studies of “normal” behavior rely on college students or paid volunteers rather than the population at large). Also, despite all the advances current technology offers, there’s still a lot we don’t know about human functioning and its myriad intricacies.
In Mr. Johnson’s case, he identified a reason for going against what is current scientific wisdom regarding raw milk (albeit one that speaks not to contamination issues, but overall digestion capabilities). But there, too, is the influence of the aggregate: government busybodies and bureaucrats have decided that raw milk is risky to us all, and since most research in this country is funded by some institution or other (gov or corporate interests), the findings are couched in language that appeases the pocketbooks.
Science, it seems to me, has become an institution in its own right—an unassailable purveyor of rock-solid conclusions, of truth. But for many reasons—and for me, its reliance on aggregate information being foremost among them—that perspective is narrow and potentially dangerous. Science is supposed to be a self-correcting way of gaining understanding; it was not intended to prescribe and proscribe things at an individual level. This view, along with the myth that a study or two proves a particular finding, has left me (a former researcher!) very skeptical of the institution.
Still, I read the news articles, and when I’m sufficiently interested I’ll try to find the original research in order to grok more fully. But—perhaps because I am one of the minority of lefties on the planet—I do not order and re-order my life around the findings of the day. I might try something, and if it works for me, great; if not, I don’t worry that I am somehow defective. I’m simply an individual—and the bulk of science is not oriented toward an individualistic understanding.
Interesting...
I think that you may have received that link from me. I put it in the del.icio.us bookmarks that go into my RSS feed since I found it interesting.
While I understand your problems with scientific methodology, I think the greater problem is with the media misinterpreting the results of said studies. I was never a scientist, but I used to analyze statistics in the electronics industry to find problems with our products and processes. One thing that I learned is that laymen (including managers who think that they're experts) do not understand the context of a particular statistic, and therefore misinterpret what it means. I believe that reporters are especially guilty of this when looking at scientific studies. Science reporters rarely have any background in the subjects that they are reporting on, making the problem worse.
The media is also loooking to get an attention-grabbing headline to sell newspapers or whatever, and often skews its interpretations to grab the reader's attention. It sells more papers when you have the headline "Scientists say that X causes cancer" rather than "Researcher X in a study at Harvard concludes that there is a 0.2% statistical increase of cancer risk after injesting Y(probably colossal) amount of substance X for Z number of years when combined with risk factors A, B, and C." This is why I tell people to not base any personal behavior change on the report of some particular study reported in the media. The interpretation presented is probably wrong, either through ignorance or deliberate manipulation of the facts.
This is especially true with hotbutton subjects such as (I am afraid to even bring it up) global warming. You see misinterpretation both deliberate and accidental in media reports that come down on both sides of the issue. A single hot day or even a single hot year does not necessarily signal a trend. Conversely, the presence of a snowball somewhere in the world or a cold winter does not eliminate the possiblity of an upward trend. One data point tells you nothing about trends either way. Many other factors also come into play and it is difficult to isolate a single factor such as "greenhouse gases." None of this matters when reporters (or even bloggers) are trying to push a particular conclusion. They want their zinger headline, and leave no room for doubt in their conclusions.
While media reports such as the above mentioned raw milk article are often interesting, when it comes to the science behind it I know to not take a reporter's conclusions too seriously. I must go to the original study to see what the researcher was actually saying, and only then can I begin to evaluate his or her conclusions and methodology.
In defence of science
I think much of the problem in science comes from societal attitudes toward it. It is viewed as either a god or a charlatan. As Presto said, too much science is misrepresented in the media by people with no background knowledge in the subject or just looking for a good headline, thus giving the idea that it is the scientists who are ignorant, not the reporters. The idea that science is an unassailable god, as Sunni mentioned, is promoted in the university system by arrogant professors who are right because some paper says they know all and are some sort of expert. The scientific theory is really not taught in school, schools tend to follow the idea that scientists are all experts or frauds. When I studied biology I had a few professors who truly understood science. We also understood science isn't fact, but only observation; that science is never static and sure, but ever changing, because our knowledge changes daily. Much of what I learned in genetics class is that we really don't know as much as the media makes out that we do. Television shows also give a false idea of what science is capable of (in fictional forensics shows especially (think CSI)). What I love about science is that lack of permanence, knowing what is taught today may prove to be completely wrong tomorrow (the Earth used to be flat), that change is the only constant.
More on science (Moron science?)
Okay, you guys have got me going again ... First, Presto:
That is certainly true, and your comments regarding sensationalism in the media are also spot on. However, there’s far worse afoot out there. When I first began teaching introductory psychology as a graduate student, I wanted to do the best job possible; therefore, I used several textbooks as resources for my lecture notes. (I never was one of those instructors who merely read the assigned textbook aloud in class.) I noticed that descriptions of classic experiments—all presented as “fact”—varied, sometimes widely, among the texts. How could that happen if the authors were doing basic fact-checking? (I began reading the original research after noticing this, but it is a time-consuming process; and it requires an excellent library.) Anyway, if the so-called experts in a field, who are writing the texts for future experts, can’t be bothered to do that kind of work, why should anyone else? Sloppiness is shot through the system—at least in my former field.
More from Presto:
You’re right, of course. However, the research process is not easy to grasp (well, that or maybe I was just slow at grokking it): although I could have taken a minor in quantitative psychology in grad school, and did lots of research throughout my educational training, it wasn’t until I began teaching experimental psychology that I deeply grokked how it all is supposed to flow—ideas into research hypotheses; hypotheses into testable questions; testable questions into a rigorous research design; collected data into inferential statistical tests; and the results of those tests back into general conclusions. Of course, there are potential land mines at every step, and plenty of places for bias (conscious or not) to seep in to the process.
These days, the “celebrity scientist” plays a role in the sensationalism as well. I highly recommend the excellent book (I ought to review it) Of Moths and Men for some insight into this development and the scientific process as well.
And white rabbit says, in part:
Of course; and this simplified dichotomy is the inevitable result of an educational system that is designed to dumb down children across all levels of ability.
One of the things I emphasized in all my teaching is that science is a method—and that method also changes over time, as does our understanding.
Your mentioning of genetics reminds me of another widespread fallacy—that our genetic code is somehow beyond influence. It is what it is and one is either genetically doomed or blessed. Environment can influence the expression of one’s genes, sometimes radically; and genes get switched on and off over the course of a life, perhaps by situations that geneticists currently don’t fully understand. It irks me to no end to see the genome portrayed as some unassailable tyrant of life.
I understand, I think, why you titled your comments “In defense of science”, but my wandering babble shouldn’t be taken as a complete repudiation of it. Science is a system—and in our current environment, it’s a system influenced by political, economic, and cultural ideas to a disturbing degree. But beyond that, the system of science is largely both reductionist (in methodology, mostly) and aggregate (in interpreting/applying results). Because of those elements, scientific findings are not easily mapped into a holistic, individualistic perspective. Yet that’s what science celebrities, media, and many others smuggle in to their articles and interviews—and I have enormous difficulties with that misrepresentation.