Josh Haas's Web Log

A modest proposal to fix science

with 13 comments

Here’s the proposal: separate hypothesis generation from hypothesis testing.

My inspiration is this great post  which demonstrates how remarkably easy it is for experimenters to produce results that support their hypothesis, regardless of whether their hypothesis is correct.

The idea: outsource all experimentation to special labs. You send them a hypothesis and a check, they send you a “disproved” or “could not disprove” paper; the lab signs, you sign, and it gets published in Nature.

Right now research professors both generate the theories and perform the experiments. In this world, they just generate the theories, and then use their grant money to pay other people to test them.

Also, companies and private individuals could use the labs as well. Which both democratizes and standardizes research.

The labs then become specialists in performing strictly controlled, statistically valid research at an efficient price-point. Since they become basically science factories, it’s much easier to audit to see if the experiments they are doing are valid, and because they’re doing thousands of similar experiments, should be able to drive price down… it’s more like running a McDonalds than running a research group.

Meanwhile, it frees up professors to actually focus on generating domain-specific insights, instead of forcing them to be experts on statistics and valid experimental techniques.


EDIT:  A couple people have pointed out that it’s the process of experimentation / getting into the nitty-gritty of things that leads to hypotheses.  Which is a great point.  So let me amend the above to say, professors and such can and should still do experimentation in their own labs.  But, if they want to then get a conclusion published in a peer-reviewed journal, they should outsource the official verification work to an external lab.


Written by jphaas

April 30th, 2014 at 2:40 am

Posted in Uncategorized