Optimism, and some answers.

There’s a new kid on the blogging block –  The 100% CI – and I 100% recommend you check them out and invite them to your birthday party.

As for me; I arrived a little late to the party today – specifically, to the party happening on PsychMAP around Julia Rohrer’s 100% CI post on (mild) optimism for the future of psychology.

A couple of empirical questions popped up at that party, and since I have some relevant data, I thought I’d post some answers here.

Empirical question 1: The difference between early- and late-career researchers.

Okay that’s not a question. But people mused on the difference, wondered how big it really was, asked whether we could “move beyond” it to other predictors of engagement with better research practices.

So in response, I can point you to the data on my OSF page (and the preprint), and tell you that:

  1. In  sample of 162 social psychologists, those with tenure (relative to those without) saw significantly more costs and less benefits to practices like direct replications and pre-registrations. The effect size was around d = .42,  which is impressive for social psych but maybe not so much for the real world? I’m personally more impressed by the overlap of the distributions; we have so much in common.
  2. We can indeed “move beyond” this dichotomy; and look at what predicts actual behavior. Which brings me to the next question…

Empirical question 2: What gets people to actually act differently?

I’m just kidding, who would ever measure actual behavior?? People’s self-report is totally reliable.

Sigh. Not-kidding though, I wish I did have measures of behavior. But this was an online study; all I have are self-reported likelihood of engaging in X in the next 6 months, on a 1-7 scale. On the whole, I have some faith that people were answering these questions sensibly – for example, they said they were more likely to make their data open (Mean = 5.02, SD = 1.78), than they were to submit a registered report to a journal (M = 2.29, 2.00).

As for what predicts these responses?

The answer seems to be… efficacy. And, to some extent, dissatisfaction with the current state of your subfield. (But not dissatisfaction with your own role/job, nor identification as a research psychologist.)

(…I was going to paste in all the tables here; but it starts to look messy. Just trust me? Or go to the OSF and run the analyses yourself.)

This result is correlational: greater efficacy –> greater likelihood of better research practices, or greater likelihood of better research practices –> greater efficacy? But I still like it because it makes me optimistic (!) about all the ways that so many smart people are making it easier to do good science. Maybe we can argue less about why we have to do good science, and move on to how to do it? I think this is part of what Anne Scheel was getting at when she said that “awareness is one thing, what about acting differently?” (I paraphrase.)

Empirical question 3: If you’re pro-change, are you more likely to leave academia?

This question might seem tangential, but I think it speaks to this weird thing where  pro-change people (i.e. almost everyone I talk to on twitter) get worried that other, less scrupulous, researchers are climbing Mount Tenure using QRPs (nice imagery, thanks Julia!) and so feel disheartened and might quit, whereas pro-business-as-usual people get worried that data police and statistical bullying are upsetting good researchers to the point that they might quit.

So. Correlation between being pro-change and self-reported likelihood of leaving academia?

Nope. At least not in this sample.

I have measures of pro-change in relation to direct replications and pre-registration separately, but in neither case is the correlation significant (N = 257).

Take home message? Nobody’s leaving; we’re all in this together! Remember our similarities as well as our differences, and think about the things we can learn from each other. I may not agree with Fiske that we’re not in a crisis, but I’d love to find out where she gets her optimism from.



3 thoughts on “Optimism, and some answers.

  1. Pingback: Five Reasons Blog Posts Are Of Higher Scientific Quality Than Journal Articles | The Global Warming Policy Forum (GWPF)

  2. Pingback: Five Reasons Why Science Blogs Beat Mainstream Journals - Principia Scientific International

  3. Pingback: Five Reasons Blog Posts are of Higher Scientific Quality Than Journal Articles | US Issues

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s