Eric Waller did a quick experiment to confirm a behavioral economics hypothesis with a small amount of Mechnical Turk data. He found a paper that showed evidence that removing the minimum payment line from a credit card statement causes people to pay more (unless they typically pay the bill in full), and constructed an experiment to confirm the hypothesis.

The entire process took him only three days. I’ll let you jump over to the article to see the results!  It’s a nice example of how quickly and cheaply a short, simple experiment can be run on Turk — Waller spent a total of 3 evenings setting up the experiment, recruiting and running 200 participants, and analyzing the results.  As we’ve discussed before, Turk is a good tool for this kind of experiment.

As this kind of research becomes easier, it also makes it more likely that people will do more research like Waller’s — confirming things that they’re pretty sure are the case, but which should really be double checked, as well as fleshing out existing results a bit more precisely.  That seems like a pretty great meta-result to us.

Our mission involves recruiting large populations of internet users for research tasks, so we’re always interested in innovative crowdsourcing methods.  Crowdsourcing services allow many people to contribute to a project and be compensated in various ways.  We stumbled on 10 ways small businesses can harness big crowds by Ross Kimbarovsky, co-founder of CrowdSPRING, a sometimes controversial marketplace for design services.

The most interesting services highlighted are software testing (uTest), customer support (the always fantastic Get Satisfaction), domain-specific scientific, materials and technology research (InnoCentive), and prediction marketplaces (Inkling). The most successful crowdsourcing projects seem to be those that offer a win for both the business and the community members.  As a company developing tools to help both researchers and participants, this makes intuitive sense to us, and we’re happy to see this strategy succeeding elsewhere.

The NIH has just announced a great new tool for clinical trials and other IRB-approved research!  ResearchMatch.org allows people interested in participating in research to sign up to learn about specific studies they might want to participate in.  It also lets researchers to use the system to recruit participants for their research.  In contrast to our online research focus, this tool fills a gap in research done in the laboratory, where it can also be very hard for researchers to recruit study participants.

Despite being administered by the NIH, ResearchMatch.org is for any IRB-approved researcher (not just clinical trials).  It contrasts in that way with research-participant match-up site ClinicalTrials.gov.  Another difference between the sites is that ClinicalTrials.gov makes the participant do the work of finding research they’re interested in participating in, while ResearchMatch.org makes it the job of the researchers to contact participants that match their needs (the system protects potential participants’ personal information however).

Currently, the site only allows researchers to use the system if their university or institution is a participating member.  But they’re encouraging researchers to sign up for information and express interest even if their institution is not currently participating; I don’t know if the site will eventually stop being mediated by institutions, or if they’re just hoping to get lots more institutions to sign up soon.

A side effect of the institution model is that opportunities are sparse in some parts of the U.S., and if you’re interested in participating in research in those regions, you may be out of luck for a while.  But I don’t know, maybe the network will grow fast — this tool seems like a terrific idea, and I hope lots of research institutions, researchers, and people interested in participating all sign up.

I also hope that if this system is successful, other countries will emulate it. For that matter, maybe the U.S. is lagging behind here and other countries already have such systems; does anyone know of such research matchmaking sites elsewhere?

If you haven’t yet checked out Amazon Mechanical Turk for doing research with human subjects, you should!  We like Turk for running short, simple studies.  For instance, if you want to find out how several hundred people would describe a single picture, or survey a big crowd to find out which name they prefer for a new product you’re releasing — and you want the data really fast, Turk is a really useful tool.

When I did my thesis work in cognitive science, one question I was interested in was how people interpreted words like “tall” and “long” and “big” in different contexts.  I was able to show thousands of people one drawing each and ask them to pick out the tall or long items in each picture.  It was a fast and inexpensive way to get data, compared to bringing people into the laboratory.

So does it work well?  Earlier this year, Language Log had a nice writeup of a linguistic sentiment experiment on Turk that they ran primarily to test Turk’s reliability for simple linguistics experiments.  They found the results to be quite good.  Another great analysis, courtesy of Dolores Labs, concludes that Turk is “fast, cheap, and good for machine learning data.”

Other questions that researchers have is who participates in tasks posted to Turk, and why.  Panos Ipeirotis has some great data on why people participate in tasks on Turk.  FloozySpeak provides more detailed information on what entices people to participate — and how much time they spend on Turk.  And Ipeirotis (a great resource on Turk in general) does a nice job analyzing the demographics and of Turkers.

Mechanical Turk was not designed with scientists or researchers in mind, nor do they have human subjects’ rights on their mind in watching out for the participants.  There are a lot of times when the system can be a bit frustrating when doing research — especially longer or more complex studies.  But it can be a great tool for many things!

Dolores Labs is doing a bunch of cool work finding ways to distribute tasks to large numbers of people online.  As well as crowdsourcing tasks, they also check the results to make sure that you get higher quality results than using unmoderated tools like Amazon Mechanical Turk.

The Dolores Labs blog has an interesting post up about their recent partnership with Gambit (a payment tool used in online games and communities).  Together, they’re offering Facebook gamers the opportunity to earn in-game credit for doing real work through their platform.  This is a timely service, especially given TechCrunch’s recent indictment of offer platforms as scams. Now you can earn your in-game wage by completing simple tasks instead of signing up for credit cards and Netflix accounts!  We like this innovative way of reaching and rewarding large numbers of people via Facebook.