The perils of volunteer bias

Hey everybody--guess what? I'm talking shop again. But how can I possibly pass up the opportunity when I see *this* over at DFA-link.

Robert,

Thank you for participating in my survey. I hope you won't be the only one!

If anyone else would like to participate, here's the link:

efm.pulseware.com.au/survey.asp?...

by Noah Porter on Friday, 10/06/06 @ 03:31 AM | Rate this | Avg Rating: -1

You'll note that I left the comment, including the rating, intact. The survey is here if you'd like to participate. Wonder how Noah's participant responded to this question:

What beliefs do you share in common with DFA?

But honestly, this is an important point, and I think it's important enough to address in a front page post. Because trying to find a reference to illustrate this, I looked up two famous examples of what is called "volunteer bias" in survey data. The first thing I thought of was Shere Hite's survery on "women and love"--where she found that the vast majority of women who responded to her survey were unhappy with their relationships, and an astounding percentage of them admitted to having affairs. Yet of the surveys she sent out, only about 4% were returned. Do you see why I bolded that one part of the sentence? We don't know anything about the "average American woman" from her survey--we only have information about the 4 percent who responded. Most of *those* women, apparently, were pretty unhappy in their relationships.

But are they typical? How many of you receive surveys of one sort or another in the mail? How many of you respond to them? What would motivate you to take the time to respond? Often, having strong feelings about the subject matter is what motivates people.

When I searched for a link about this study, I found a lot of reports about volunteer bias in *sex research*. But the problem of survey respondants differing from "average" member of the population under study is by no means limited to research on human sexuality. So I looked up another well-known example--the one where Ann Landers asked mothers if having children had been "worth it", and 70% of those who responded said "no". Unfortunately, most of what I found for *that* survey involved people using it to make the point that people often regret having children. (Though they usually paid lip service to the notion that "of course this survey is unscientific"...but look at these numbers!)

This is from an article in Salon entitled "To breed or not to breed":

"It's very rare for a woman who has children to regret having children," says Hanson. "You will find women who say, on the one hand, 'I love my children, they're profoundly fulfilling and I can't imagine not having had them.' But on the other hand they'll say, 'Boy, this is really stressful. This has really strained my marriage. My health has never been up to par since I had my first or second child. I really regret the impact of having kids on my career. Having children has made me financially dependent and really limited my options for making money.' I hear them say all those things, but you rarely hear moms actually saying, bottom line, I should never have done it."

Yet Cain insists that more women feel that way than might admit it. She points to a famous survey advice columnist Ann Landers took of her readers in 1975. A woman wrote to Landers with qualms much like mine -- she and her husband were torn about childbearing and asked, "Were the rewards enough to make up for the grief?" Landers put the question to her readers, asking, "If you had it to do over again, would you have children?" Astonishingly, 70 percent of her respondents said no.

"Seventy percent of whom?" I might ask my students. Of American mothers in the 1970's? No. Of American mothers in the 70's who read the Ann Landers advice column? No again--although note for a moment that the demographics of regular readers of Ann Landers are already likely to be a pretty big step away from the broad category of "mothers in America" in that time period. No, we are looking at 70% of the mothers who read that survery question and felt *motivated to respond* for one reason or another.

I couldn't find an article addressing the issue of volunteer bias in this particular survery, but I did find this page with answers from old multiple choice psych exams. I've seen similar questions in the test banks I've used. It's an important concept for students to understand.

Anyway, here's that link again for Noah's survey, if you consider yourself to be a member of DFA and would like to help give a more balanced picture.

Alternate link for comments