I love anecdotes. I sprinkle them liberally in my writing. I know, as a voracious reader, that stories make ideas come to life. How can I illustrate the idea that some people do a lot with their time? I can tell a story of a woman who owns a small business and has 6 kids and still creates me-time in the middle of a busy Thursday morning. How can I illustrate the idea that, at least for middle-income people, money is about choices? Profile a young couple that lived modestly for 5 years, then traveled the world for two. The human brain loves coherent stories and hence treats them as saying something broader about the world. This is how we are wired.
The problem is that while this wiring is fine for being entertained around the campfire, and I obviously choose to use it (either to dramatize data or to give a counter-factual that shows a universal statement must not be true), it’s trouble for making serious decisions. Because anecdotes aren’t data. Indeed, stories on their own are pretty much meaningless.
Here’s why: there are 7 billion people on this planet. In a world of 7 billion people, you can find anecdotes of just about anything. You can find people who were raised by saints and became criminals. You can find people who were raised by criminals and became saints. You can find people who exercised daily, never smoked, ate organic kale for breakfast and dropped dead of a heart attack at 35. The fact that you see something in your life, or your neighbor sees something, does not mean that someone else isn’t seeing exactly the opposite, or that if any given person did what you are doing in your life, they’d see the same result. Beyond that, the human brain has a tendency to form narratives, and then look for evidence to support said narratives. So none of the 7 billion people on this planet is dispassionately observing every bit of evidence coming at him or her. We see what we want to see.
To be sure, this doesn’t mean we must be like Pontius Pilate, asking “What is truth?” Many smart people spend their lives trying to study human behavior and human outcomes. Often times these studies try to get data that is not so influenced by perception. You did eat a certain number of calories yesterday. You did sleep for a certain number of hours, drive a certain number of miles, and children at your local school did get certain scores on an internationally benchmarked test. Sometimes data are wrong or misleading. But over time, as the data add up from multiple studies, we can start to know things. Sometimes these things are very different from our stories!
For example, the American Time Use Survey relies on time diaries to draw conclusions on how Americans spend their time. Audits of time diaries find this methodology is more accurate than simply asking people how many hours they spend doing X, Y or Z. According to the 2011 ATUS, the average American sleeps 8.71 hours in a 24-hour period. Whenever I report this number, though, someone invariably tells me “That can’t be right! I don’t sleep 8.71 hours!” This is not an argument. An average means nothing about you. Even if you go on to say that “no one I know sleeps 8.71 hours a night” this is still not an argument. None of us knows a perfect cross-section of Americans. And even if we did, we likely aren’t there watching what time they go to sleep and wake up every night, including weekends and holidays. Instead, we are relying on anecdotes from friends, in a world where things that suck (e.g. sleepless nights) are remembered more vividly than things that don’t.
Unfortunately, this distinction between anecdotes and data doesn’t always show up in journalistic reporting, blog writing, etc. Three stories make a trend! Though I guess it’s better than using fake data, which is an entirely separate issue.
What’s your favorite incidence in which an anecdote has been given more weight than it should have?
Don’t the American time use surveys assume you’re doing one thing at a time and bother to write it down? Are there any surveys where you wear a data tracker? I really question whether people record every time they spend 5 or 10 minutes looking for one of their children’s shoes. Is this free time? Quality parenting time? Need some new children time?
My favorite quote on anecdotes is “The plural of anecdote is not data.”
LOL at “need some new children time” 🙂
As someone who voluntarily keeps time logs from time to time to help tighten up my time usage- it really isn’t as intrusive as you make out. My anecdote on this is that it provides a far more accurate view of how I’m using my time than just asking me to say how much time I spend working, for instance. And the actual data on that agrees with me!
Anecdotes always seem to trump findings when people talk about the medical literature. If a study shows clearly that a screening tool isn’t worthwhile in a population of patients, there will nonetheless be people eager to tell the story of their aunt’s brother-in-law whose life was saved by that screening. It doesn’t matter how many studies show that planned attended out-of-hospital birth has outcomes that are statistically indistinguishable from in-hospital birth (this is only true for low-risk women with singleton vertex babies), someone will always respond by saying, “My baby and I would have DIED if I hadn’t been in the hospital.”
@Jamie- very true. And as a result, there are physicians practicing medicine by anecdote. The logic all this seems to be using is that the worst outcome of not screening is worse than the worst outcome of excessive screening, but I’m fascinated by how many people never take into account type 2 errors. For instance, if you’re in a hospital, even needlessly, you still risk acquiring a nosocomial infection. If you have a biopsy that wasn’t needed due to aggressive screening, you risk errors in the procedure, side effects, infections, etc.
I also went and found data for “middle income” couples (middle quintile of household incomes) is $36-$57k. The 80th percentile (more dual income couples) is $91k. While money is partly about choices at all income couples, both of these families will find their choices (about family size, education, savings, outsourcing) pretty severely. I think you’re using “middle income” to mean “top 5-10% of incomes” again- shame on your statistics teacher. 🙂
Bad typing, sorry!
also went and found data for “middle income” couples (middle quintile of household incomes) is $36-$57k. The 80th percentile (more dual income couples) is $91k. While money is partly about choices at all income levels, both of these families will find their choices (about family size, education, savings, outsourcing) pretty severely constrained by lack of money. I think you’re using “middle income” to mean “top 5-10% of incomes” again- shame on your statistics teacher.
In fairness, I’ve seen anecdotes of couples and even families in those middle income ranges (even the low end) uprooting themselves to live in an RV and travel across the country (or even across another country). It’s something one runs into occasionally reading public finance blogs. Common, probably not, but that’s the point, right?
@Twin mom – of course more money allows more choices, but average households — budgets of $49.7k, with 2.5 members, per the Consumer Expenditure Survey — can still make plenty of choices. After all, we had the whole debate about the family earning $460/week with 6 members, and they were still making some choices, albeit very constrained ones. As one example of such choices, cell phones have become ubiquitous over the past 20 years, and are now a regular part of middle-income budgets, but since we survived without them 20 years ago, this implies they’re not an absolute necessity. People with drastically different beliefs than mine write about the choices available at modest incomes. I really didn’t like that book on Radical Homemakers, but it was all about the ecologically-friendly lifestyle choices one could make on family incomes of less than $40k/year. These anecdotes aren’t data, but if you’re trying to argue that true middle income families, as opposed to the ones my allegedly mathematically-addled mind think are middle income, are prisoners of circumstance, then anecdotes of varied lives can counter that.
I think you underestimate the degree to which families which incomes of ~$50k are constrained in their choices. I think your anecdotes look at the successes vs. the challenges (need to support elderly parents, children with disabilities or high medical bills) or even the average (flaky boyfriend, student loans)
People with incomes of $50k probably have cell phones, but often in place of a landline, which I view as a substitution for what would have existed a couple decades ago.
I see anecdotes being used to argue against data a lot when people talk about sexism. In fact, I’m disturbed to think that I probably do it, too. Did you read that Forbes piece I linked to last week, about how women like me will say that the pay gap has not impacted us, when the data says that it almost certainly has? (http://www.forbes.com/sites/shenegotiates/2012/10/01/is-it-crazy-to-assume-the-pay-gap-doesnt-apply-to-us/)
It happens in much more insidious ways, too- for instance with the “well, SHE has succeeded so there must not be any sexism in this industry anymore” trope.
Ugh. Hate to open THIS can of worms, but the whole “vaccine-autism” thing is a perfect example. Despite repeated evidence to the contrary, there is always “my second-cousin’s roomate’s brother’s child”. Or, as Jamie mentioned above, really any evidence-based medical practice. There was the finding recently that older fathers are associated with higher rates of autism—and the comments (why are there comments enabled on an article about medical studies?) were basically “well I had my son at age 50 and he is a genius and most popular in his class”. Or the one about antibiotic use associated with increased obesity “well my daughter got antibiotics four times for ear infections and she is very skinny”. If this kind of preliminary correlative data is going to be presented to the general public, then all schools should start teaching the basics of how research studies are conducted, correlation vs causality, and critical evaluation of the literature.
Sadly many medical researchers don’t know research basics either. (Not saying most, just that when, anecdotally, I assign my students to bring in articles showing that the author doesn’t understand the difference between correlation and causation, the majority of the articles are either medical or education. How some things get published is beyond me.)
@Ana- yep, that’s a can of worms. There’s also the fun of lying with statistics. Raising something from a 1 in 100,000 chance to a 2 in 100,000 chance amounts to a 100% increase, which people sometimes choose to hear as a 100% chance of something happening. So ban it now!
Ooh, we have a post on that too… http://nicoleandmaggie.wordpress.com/2012/05/29/percentage-vs-percentage-point-a-primer/
Data analysis and simple stats are something that should be taught in high school, and I’m always surprised that they’re not. (Maybe the really good private/magnet schools do?)
I managed to get both a B.S. and an M.S. degree in biochem without ever having to take stats (shameful!), except for a cursory overview in a genetics class.
This is another “anecdata” story, which I think is why it is so damn interesting to read:
http://www.reddit.com/r/TwoXChromosomes/comments/hvv2m/i_work_for_a_large_multinational_tech_company_i/
@ARC – I think that’s a good use of anecdote to illustrate data that we’re pretty certain of. There is a pay gap. Women are also less likely to negotiate than men (another data point that’s been studied). This gentleman is illustrating his own experience with a story that makes those data points come to life. A bad use of anecdote would be to say “pay discrimination doesn’t exist because I just hired a man and a woman at the same salary.”
As a former vegetarian who converted to a meat-based diet, I subscribe to news feeds from both camps.
One of my favorite examples of anecdote are memes generated by both camps. One will feature a healthy-looking leader of the vegetarian camp, compared to an obese leader of the meat-eater camp. Who would YOU want to be like? The same meme shows up in the meat-advocating news feeds (but reversed of course).
While we are on the topic of lying with statistics, what about the statistic of the chance of birth defects doubling when a woman has children after a certain age (usually after age 30)? People are horrified when I talk about having a kid at age 35 or 36 (I’m 34). That number is still really small, but people still blow it up for some reason. We really need to become more literate on statistics…a 1 in 100000 chance or a 2 in 100000 chance are still both really tiny!