Friday 11 January 2013

Peerless honesty



Alright, hands up. Who’s been consoling themselves after the return to the office (happy New Year, by the by!) with the Twitter hashtags, overlyhonestmethods and overlyhonestreivews? If you haven’t, may I recommend that you do so at once.

Are you back? Ok. The overlyhonestmethods tag, started by one bored postdoc with the post-Christmas blues, has grown, very fast, into a sort of mass online confessional. I’m sure, like me, you winced with recognition at some of the more candid reasons behind methodological choices - the sampling ones were particularly close to the bone when I think about my Masters dissertation. But I’m proposing to focus in this post on the overlyhonestreviews tag, and (gulp) put down some of my thoughts about peer review. 

Not to pat my own back, but I think this is pretty brave of me. As the scholarly communications system swirls with new ideas - business models, platforms, what it is that we even publish - peer review seems to be something of a lodestone. A fixed point that we can always return to, a characteristic of scholarly communications that simply cannot change. Researchers, in particular, are very keen on peer review, and in many of the open-access-type discussions I've been attending recently this has been a big theme. Peer review is non-negotiable.

And I think that the overlyhonestreviews tag on Twitter suggests some reasons why this uncritical approach mightn't be the right one. Let's start by saying that these 140-character reviews are funny because they contain a little grain of truth. I imagine we've all read papers (unpublished, and published too) where we've found the methods peculiar, the literature review partial at best, and struggled to see the connection between the modest results obtained and the spectacular conclusions drawn. These author-side problems are all highlighted in the stream of comments, with considerable humour in some cases. And they show peer review working as it should.

But there's another set of comments, with (I suspect) equal levels of truth, which is much more of a concern. A few media stories on overlyhonestmethods have picked up the theme that 'scientists are human too', and as we know, humans are not Made of Nice. This comes out in a subset of joke reviews which reveal the pettier aspects of the current system. You know what I'm talking about here. 'It's pretty good but you haven't cited me - REJECT'. 'Oh dear. You appear to have pre-empted the work I'm doing at the moment, making it virtually unpublishable - REJECT'. 'I met you once at a conference and you were annoying - REJECT'. 'This method was devised after 1990 so I've never heard of it - REJECT'. Or even 'This is a bad piece of science but you cite me and that's good for my h-index - ACCEPT!'. Others reference the power relations in research: 'This paper's so good that if I let it through you might become a threat to me - REJECT'. 'This is a pile of rubbish but I'm a first-time-reviewer who's slightly scared of you, so I'll ACCEPT'.

I'm not suggesting that any of this is routine, or even particularly common in scholarly publishing. There's a lot of exaggeration for comic effect. Nonetheless, that niggling grain of truth remains. Particularly when you consider that the outcome is most likely not an outright 'reject' but a round of time-consuming revisions to include the reviewer's extensive back catalogue, or to explain the method in words of one syllable to readers who have been using it for years. That's not a system which is functioning efficiently.

Of course, we're aware that peer review isn't perfect. Studies have looked at whether specific groups - young people, women, ethnic minorities - are unfairly disadvantaged. And there's been a long debate around the relative merits of single-blind, double-blind and open peer review. But I'm not aware of any studies which look at the underlying process of peer review - whether the attributes identified in the twitter stream are nonexistent, niche or widespread, and what that means for the content that makes up, broadly speaking, the scholarly literature.

You might argue that there are some experiments underway which could help move things on. Initiatives such as PLoS One, the altmetrics movement and a current study at the University of Tennessee looking at how researchers assign authority to scholarly outputs are all working within the same territory as peer review. To that, I would say that these experiments are primarily about assessing significance rather than quality. Most are working with outputs that either haven't been quality assured at all (blogs, for example etc) or those which have been formally published (journal articles) and have therefore already been through peer review to check they're technically sound. Indeed, this is the key aim of PLoS One, and the area where altmetrics probably have the strongest claim to usefulness (for now, at least).

So, we must assume that there are some - not many, but some - articles which are technically sound, but which don't ever make it through the peer review process to post-publication judgement because they've come up against the barriers outlined in the stream of tweets. And, that there are some - fewer, probably - articles which aren't technically sound but do make it through the peer review process, to be judged by readers only on their significance while the quality is taken as read.

Perhaps this isn't a great revelation. But my strong impression, on reading what started out as a lighthearted distraction from work, is that we really should look beyond the jokes to talk about this seriously. As governments, funders and researchers themselves press for changes in the way research is communicated - including sharing new types of work such as datasets - we should seize the opportunity to ask whether peer review is the only - or, indeed, the best - way to ensure that scholarly communication works effectively. I'm not saying that we have the answers. But we never will, unless we overcome our peer review taboo and start to ask the question.