keeping participants honest in online research

Let’s face it — research participants sometimes lie, cheat, or just don’t pay attention.  This can be a problem in the laboratory; I was at a dinner party the other night where some researchers were discussing recent work done showing that as many as 30% of psychology research participants in the laboratory are not paying attention to instructions (and presumably are just trying to get paid quickly, in many cases). [1]

But it seems natural to expect that this problem would be magnified even further online, where people are unsupervised and anonymous.  Part of the work that HeadLamp Research intends to do is to investigate how reliable the data collected on our online platform are, and to look for ways to improve reliability.  A first step, though, is brainstorming why and how participants  might be dishonest in the first place, so that we know what to look for.

Here are some of the things that seem like particular worries to us with online research:

• Participants may register multiple accounts to participate in studies more than once.
• Participants may lie about their native language, age, or other personal information in order to be eligible to participate for more studies or for better paying studies.
• Participants may lie about their personal information like education or health background because they are embarrassed to tell the truth.
• Participants may fail to follow instructions and simply get through a study as quickly as possible in order to maximize their pay per time.

What are we missing?

We’re not the first people to be looking into deception online, or even the data reliability of online research.  So there’s a research base for us to build on.  And we have some ideas of our own about how to detect liars, cheaters, and those who just aren’t paying enough attention.  We’ll be talking about this more, but we’d love to hear how you deal with these problems in the lab, and what your major concerns are in terms  of data reliability.  In some of my research, it’s been essential to have participants with particular linguistic backgrounds.  What factors are most important to your research?

(And, by the way, for those research participants out there — we’ll also be talking about how to keep researchers honest; we know they can also occasionally screw up or be unfair, and participants should have a way to deal with that, too!)

[1] I don’t have a citation for this yet, unfortunately; it wasn’t clear to me if this research had been published yet, but I’ll be looking it up.