Let’s say you’re trying to get a sense of how the U.S. population is going to vote in the upcoming presidential election.  So you take a poll of 100 people that are your friends on Facebook.  You are excited to find that the candidate you intend to vote for is going to win in a landslide.

Are these results a reliable prediction of the next election?  No, they’re probably not.  Your friends who have access to Facebook are almost undoubtedly not a representative sample of the U.S. population as a whole.  Doing social science research online carries a similar risk: if you’re trying to generalize about a large population — like, say, all of humanity — based on the population of people who have access to and interest in an online study, then you may be drawing incorrect conclusions.

This problem is not unique to online research.  In fact, some laboratory research in fields like cognitive psychology essentially assumes that all human beings’ minds function in essentially the same ways, and many researchers study a population consisting mostly of college students and/or local residents and passersby.  There are many reasons to believe that this is often a bad policy.  (To be fair, many researchers do also do significant work to try to determine which demographic factors may affect their results, and to try to adjust their sample or the breadth of their conclusions accordingly.)

So how does the online situation compare to the offline one?  Some research has been done on the demographics of Mechanical Turk participants.  This work makes it look like the online population will be, along some dimensions anyway, far more diverse than the local population that is easily accessible for laboratory work.   However, everyone participating in online research will necessarily have access to a computer, the internet, and have the free time, interest, and computer skills necessary to participate in such tasks.  This means that the online participant population will be unsuitable or insufficient for the purposes of some studies.  Still, we hope that we can provide access to a population that can speed the gathering of at least a portion of the data for some studies.  And for some studies that are currently run entirely on undergraduate and/or local populations, we hope to dramatically increase the diversity of the populations studied.

We don’t expect that online research will be able to completely replace field work and laboratory work for a number of reasons… at least, not until we’re all plugged into the giant planet-wide hive mind in the future.   But we think it could dramatically improve collection of data for a lot of human subjects research.  The fact that the NIH is trying to match up researchers and participants online for clinical trials and other lab-based research indicates that they also see some major potential for online participants in spite of potential sampling bias problems. But we’re curious what you think. Do you think you could move all or some of your research online?  What issues would you be most worried about in terms of sampling bias?  Can you think of any tools or information that would help?

Let’s face it — research participants sometimes lie, cheat, or just don’t pay attention.  This can be a problem in the laboratory; I was at a dinner party the other night where some researchers were discussing recent work done showing that as many as 30% of psychology research participants in the laboratory are not paying attention to instructions (and presumably are just trying to get paid quickly, in many cases). [1]

But it seems natural to expect that this problem would be magnified even further online, where people are unsupervised and anonymous.  Part of the work that HeadLamp Research intends to do is to investigate how reliable the data collected on our online platform are, and to look for ways to improve reliability.  A first step, though, is brainstorming why and how participants  might be dishonest in the first place, so that we know what to look for.

Here are some of the things that seem like particular worries to us with online research:

• Participants may register multiple accounts to participate in studies more than once.
• Participants may lie about their native language, age, or other personal information in order to be eligible to participate for more studies or for better paying studies.
• Participants may lie about their personal information like education or health background because they are embarrassed to tell the truth.
• Participants may fail to follow instructions and simply get through a study as quickly as possible in order to maximize their pay per time.

What are we missing?

We’re not the first people to be looking into deception online, or even the data reliability of online research.  So there’s a research base for us to build on.  And we have some ideas of our own about how to detect liars, cheaters, and those who just aren’t paying enough attention.  We’ll be talking about this more, but we’d love to hear how you deal with these problems in the lab, and what your major concerns are in terms  of data reliability.  In some of my research, it’s been essential to have participants with particular linguistic backgrounds.  What factors are most important to your research?

(And, by the way, for those research participants out there — we’ll also be talking about how to keep researchers honest; we know they can also occasionally screw up or be unfair, and participants should have a way to deal with that, too!)

[1] I don’t have a citation for this yet, unfortunately; it wasn’t clear to me if this research had been published yet, but I’ll be looking it up.

Today I follow up on the rest of the survey data I started analyzing in the last post.  We’ve gotten a number of new survey submissions since my last post; thanks to everyone for your data!  Since this post is a continuation of the last one, I’m going to keep analyzing the same set of responses from before — but the new responses don’t significantly change the results I’m presenting here, in any case.

Last time we talked about researchers’ specific needs in terms of experimental/research design.  This time, I’m going to cover the other aspects of running studies — participant recruitment and payment in particular.

» Continue reading “what researchers want, part 2″

We recently asked a bunch of researchers who work with human participants to tell us about their research needs.  (It’s not too late to give input!  Take our short survey here.)  We love data, and we know a lot of you also love data, so we wanted to share some of our results!

At the time of this analysis, we’ve had 72 complete or nearly complete survey responses, as well as many partial responses.  The breakdown by (not disjoint) fields of research is:

• 38 psychologists/related cognitive scientists
• 21 linguists
• 19 computer scientists
• 10 neuroscientists
• 9 people in other fields (marketing, political science, economics, education, evolutionary biology, human-environment geography)

You can see here the sample bias that our own background led to in our initial research!  Especially if your field is underrepresented in the above list, we’d love to hear about what you do.

» Continue reading “what researchers want, part 1″