A while ago I (among 1576 other researchers!) filled in a survey for Nature, about my experience around reproducing/replicating scientific research. A little while after that again, I got an email from Monya Baker, asking if she could ask some follow up question about one of my responses. I said yes, she gave me a call (all the way from the US!), we had a chat, and – ta-da! – the result is this paragraph in the article reporting on the survey:
“One of the best-publicized approaches to boosting reproducibility is pre-registration, where scientists submit hypotheses and plans for data analysis to a third party before performing experiments, to prevent cherry-picking statistically significant results later. Fewer than a dozen people mentioned this strategy. One who did was Hanne Watkins, a graduate student studying moral decision-making at the University of Melbourne in Australia. Going back to her original questions after collecting data, she says, kept her from going down a rabbit hole. And the process, although time consuming, was no more arduous than getting ethical approval or formatting survey questions. “If it’s built in right from the start,” she says, “it’s just part of the routine of doing a study.” “
Of course there is more to the article than that (I especially like the overview of replication attempts, and attempts at having them published), and you should read the whole thing.
In this post I’ll just add a couple of follow up thoughts of my own.
Getting out of the Ivory Tower
If you’re a researcher, I’m sure you have had some thoughts about how to interact with “the outside world”. Is it a good idea to have a blog? Should I tweet about my study? What does the PR office at my university actually do? How does this kind of writing and engagement get taken into account when I’m looking for a job? All important questions, and ones I recommend talking, thinking, and reading about. The only point I’ll make for now though, is that research-reporting, like research-doing, varies in quality. I’ve been interviewed well (by Monya), and I’ve been interviewed poorly* (here). Familiarizing yourself with the different outlets, different writers, different styles, will help you make decisions about who to engage with, and how, once they come knocking on your metaphorical door.
Science Reporting is Hard Work
I’m just going to repeat myself, because I think it is worth it: science reporting varies in quality. This is no doubt partly because it is hard work. As an example, Monya and I chatted for at least half an hour, and from that half hour, 76 words ended up in the article. The article was overall ~1500 words long. You can do the maths. Again, I think having an appreciation for the work that goes into science reporting, will help (everyone involved) once you’re in the position of having your own research written up for the general public.
About that “Rabbit Hole”…
So far, I have pre-registered one study. And so far, I still stand by the idea that although it was somewhat time consuming, it wasn’t any more arduous than any of the other “paperwork” that goes into doing research. The rabbit hole I was referring to is in some ways more important, though, because it has to do with exploratory analyses. Exploratory analyses and confirmatory analyses both have roles to play in the research process, of course. Now that I’m writing up this study, in addition to the pre-registered hypotheses and analyses, I have some exploratory analyses to report. For me though, exploring my data – that is, running analyses that I hadn’t originally planned to run, because an idea occurs to me or because I’m curious about why this or that effect popped up – can feel a lot like going down a rabbit hole. A rabbit hole with multiple forking paths. Being able to turn back to the pre-registered analyses has helped in two ways: 1) those analyses are confirmatory, which is good in itself, and 2) those analyses function as a kind of “north star”; a fixed point to which I have to relate everything that comes after. It’s impossible to get too lost, when that pre-registration is there marking the way. This second function was unexpected to me, but definitely definitely positive – and I think it will have a positive effect on my manuscript, too.
(This study has also been one of the trickiest to write up. Not because it was pre-registered; mostly because it was (is!) a study in which I asked psychologists about their perspectives on (solutions to) the “crisis” in (social) psychology. You may have participated in it, in which case, thank you! Results are on their way. It’s just so meta – how do I write to an audience of psychologists about themselves?? Further updates to follow soon…)
*I have to take some blame for the “poor” interview, of course – I was nervous, confused (I thought I would be talking with Melanie Joy, not before her), and inexperienced – but the point is that there are ways that the interviewer could have improved the interview, and there are ways that I could have been better prepared: e.g. by listening to more of her shows in advance, so I’d be aware of her rather abrupt style.