Josh Haas's Web Log

A modest proposal to fix science

with 13 comments

Here’s the proposal: separate hypothesis generation from hypothesis testing.

My inspiration is this great post  which demonstrates how remarkably easy it is for experimenters to produce results that support their hypothesis, regardless of whether their hypothesis is correct.

The idea: outsource all experimentation to special labs. You send them a hypothesis and a check, they send you a “disproved” or “could not disprove” paper; the lab signs, you sign, and it gets published in Nature.

Right now research professors both generate the theories and perform the experiments. In this world, they just generate the theories, and then use their grant money to pay other people to test them.

Also, companies and private individuals could use the labs as well. Which both democratizes and standardizes research.

The labs then become specialists in performing strictly controlled, statistically valid research at an efficient price-point. Since they become basically science factories, it’s much easier to audit to see if the experiments they are doing are valid, and because they’re doing thousands of similar experiments, should be able to drive price down… it’s more like running a McDonalds than running a research group.

Meanwhile, it frees up professors to actually focus on generating domain-specific insights, instead of forcing them to be experts on statistics and valid experimental techniques.


EDIT:  A couple people have pointed out that it’s the process of experimentation / getting into the nitty-gritty of things that leads to hypotheses.  Which is a great point.  So let me amend the above to say, professors and such can and should still do experimentation in their own labs.  But, if they want to then get a conclusion published in a peer-reviewed journal, they should outsource the official verification work to an external lab.


Written by jphaas

April 30th, 2014 at 2:40 am

Posted in Uncategorized

  • anon

    Don’t title it A Modest Proposal unless you’re joking!

    • Josh Haas

      Well, I think I’m joking… I don’t honestly think this will ever happen, since currently university research is subsidized by the false hope of grad students and postdocs, so I have to imagine outsourcing it would be much more expensive

  • Corey Yanofsky

    Something like this already exists in biotech/pharma private industry, conjured into existence by economies of scale and regulatory agencies’ preference for arms-length supporting research.

    • Josh Haas

      Yep, although I wonder how independent the research really is… the way I’m imagining it, there needs to be a whole culture of good research around this where CROs live or die by their reputation as objective, whereas in pharma I don’t know if that culture really exists.

      • Corey Yanofsky

        All CROs want a broad and growing customer base to insulate the business against the risk of losing any particular client, so they devote a fair bit of resources to marketing and sales. If one customer comes to dominate the CRO’s bottom line, that could cause a problem with independence — but that would only happen if there was a larger problem of gross mismanagement of the kind that inclines investors to fire and replace C-level executives.

        This being understood, CROs do live and die by their reputation for providing useful data to clients — and faked data are not useful. For example, the company I work for was audited by a prospective client because the client’s previous CRO had been discovered to have provided faked data, and the client was once bitten, twice shy. (We passed.)

        The regulatory agencies require data to be collected according with a standard called Good Laboratory Practice (GLP). GLP standards require an audit trail be generated for any data set. No CRO wants to have GLP noncompliance result in the shitcanning a client’s application…

        • Josh Haas

          Cool, so how do you think the results obtained by CROs compare to the results contained by universities, etc. in terms of likelihood of false positives?

          Amusingly, the GLP wikipedia article’s criticism section says “Some researchers make the argument that since studies that do not meet these quality standards may be published in peer reviewed scientific journals, good science may be performed without GLP compliance.” Which is sort of the same point we’re talking about, in reverse…

          • Corey Yanofsky

            Oof. That’s not a simple question to answer; I can only really tell you about what I do as a biostatistician doing analyses and writing reports in one CRO: deliverables are set out in the contract, analyses are laid out in analysis plans before the data are in hand (a mutually-approved plan is usually one of the deliverables), and a good deal of care is taken to account for multiple comparisons, in my shop, at least. In any event, I would say that the false-positive/false-negative distinction is symptom of “dichotomitis”, a mental disorder that causes researchers and journal editors to put results into one of two bins labelled “discovery” and “non-discovery”.

            GLP is a very exhaustive standard, and good science can be performed without GLP compliance — only CROs that want a slice of the regulatory application market bother with it. There’s a lot of commercial science prior to the push to get a product to a regulated market. That said, the notion that peer review is a guarantee of good science is indeed laughable.

            GLP-style, but less stringent, laboratory record-keeping is a necessity for good science. The problem is that it’s time-consuming and doesn’t produce immediate scientific “returns”, so the temptation to be sloppy in the cause of “getting on with the science” is very strong.

      • Paul Torek

        How about requiring the submitter (e.g. pharma company, university) to throw in a few phony hypotheses? (And the FDA / another university could act as middleman to blind the CRO as to which hypotheses are wild and crazy.) If too many of the phony hypotheses come out “verified”, you know something is wrong.

  • Will

    Pretty sure that the creation of “science factories” will diminish the quality of the average scientist. People go into science for the promise of being able to follow their own ideas- its pretty much the only incentive left (the pay is low-especially adjusted for education, the career uncertainty is horrid- I got a physics phd at a top 5 school, not a single person in my cohort is still in science- there just aren’t any jobs). Science is already suffering massive brain drain from the more lucrative paths of finance and software, would removing the last vestiges of professionalism REALLY save it?

  • Dan

    You could have two types of journals (or two different sections of each journal), one for generative/exploratory research and one for verification/confirmatory research. All the sexy new findings go into the generative journals, but no one really believes the idea until it has been tested independently by verification researchers and published in a confirmatory journal.

    Basically, you’d be making something like replication into a high status form of research.

    • Josh Haas

      It’s going to be hard to make replication high-status, since the whole point is that it’s very proceduralizable, but, if journals start demanding replication before publishing a paper (which honestly doesn’t seem like a crazy thing to demand), then it could end up being reasonably profitable / stable to be a professional replicator.

  • Taj

    There’s a big incentives problem here, analogous to that of the credits ratings agencies. Researchers are much more likely to carry on doing business with labs that always give them good news. You could legislate it away by requiring researchers to cycle through all the qualified labs, but then think of the inefficiencies.

    Also I agree with the anon! It’s not a “modest proposal” unless you’re suggesting, say, summary execution of scientists whose results aren’t reproducible.

    • Josh Haas

      If they start with the summary executions, I’ll let you take the credit for it 🙂

      I agree the ability for researchers to choose which labs to give their business to creates incentive problems, but I think those incentive issues are an order of magnitude easier to manage than incentive problems caused by researchers doing the work in-house. With outsourced labs, you can have a well-defined API between researchers / labs, and have comparative metrics between different labs, so it’s easier to see if one lab is always giving researchers the results they want to hear. With credit rating agencies, the customer base was very homogenous — a few big banks — whereas there’s thousands of different groups doing research across the country, so I don’t think it would get as incestuous.