There’s a new kid on the blogging block – The 100% CI – and I 100% recommend you check them out and invite them to your birthday party.
A couple of empirical questions popped up at that party, and since I have some relevant data, I thought I’d post some answers here.
Empirical question 1: The difference between early- and late-career researchers.
Okay that’s not a question. But people mused on the difference, wondered how big it really was, asked whether we could “move beyond” it to other predictors of engagement with better research practices.
So in response, I can point you to the data on my OSF page (and the preprint), and tell you that:
- In sample of 162 social psychologists, those with tenure (relative to those without) saw significantly more costs and less benefits to practices like direct replications and pre-registrations. The effect size was around d = .42, which is impressive for social psych but maybe not so much for the real world? I’m personally more impressed by the overlap of the distributions; we have so much in common.
- We can indeed “move beyond” this dichotomy; and look at what predicts actual behavior. Which brings me to the next question…
Empirical question 2: What gets people to actually act differently?
I’m just kidding, who would ever measure actual behavior?? People’s self-report is totally reliable.
Sigh. Not-kidding though, I wish I did have measures of behavior. But this was an online study; all I have are self-reported likelihood of engaging in X in the next 6 months, on a 1-7 scale. On the whole, I have some faith that people were answering these questions sensibly – for example, they said they were more likely to make their data open (Mean = 5.02, SD = 1.78), than they were to submit a registered report to a journal (M = 2.29, 2.00).
As for what predicts these responses?
The answer seems to be… efficacy. And, to some extent, dissatisfaction with the current state of your subfield. (But not dissatisfaction with your own role/job, nor identification as a research psychologist.)
(…I was going to paste in all the tables here; but it starts to look messy. Just trust me? Or go to the OSF and run the analyses yourself.)
This result is correlational: greater efficacy –> greater likelihood of better research practices, or greater likelihood of better research practices –> greater efficacy? But I still like it because it makes me optimistic (!) about all the ways that so many smart people are making it easier to do good science. Maybe we can argue less about why we have to do good science, and move on to how to do it? I think this is part of what Anne Scheel was getting at when she said that “awareness is one thing, what about acting differently?” (I paraphrase.)
Empirical question 3: If you’re pro-change, are you more likely to leave academia?
This question might seem tangential, but I think it speaks to this weird thing where pro-change people (i.e. almost everyone I talk to on twitter) get worried that other, less scrupulous, researchers are climbing Mount Tenure using QRPs (nice imagery, thanks Julia!) and so feel disheartened and might quit, whereas pro-business-as-usual people get worried that data police and statistical bullying are upsetting good researchers to the point that they might quit.
So. Correlation between being pro-change and self-reported likelihood of leaving academia?
Nope. At least not in this sample.
I have measures of pro-change in relation to direct replications and pre-registration separately, but in neither case is the correlation significant (N = 257).
Take home message? Nobody’s leaving; we’re all in this together! Remember our similarities as well as our differences, and think about the things we can learn from each other. I may not agree with Fiske that we’re not in a crisis, but I’d love to find out where she gets her optimism from.