What I take from my time at JCPP
It’s been a tricky couple of weeks; after many years I have sadly resigned as an editor at JCPP. I have great fondness for the journal; it is one of few general psychology journals that regularly publishes high quality papers on developmental language disorders. Classic JCPP papers were the bedrock of my doctoral work and have shaped my thinking about developmental disorders and their broader impacts. I have published many of my best papers within its pages. I made it my mission to become an editor and remember joining the 50th anniversary celebrations, invited as a star reviewer. Being an editor has been a huge honour and will remain a career highlight. So why did I leave?
I’m on my second 3-year term as a joint editor. The first three years were a steep learning curve and it was all I could do to keep up with the 40-odd papers a year I read, find reviewers for, and craft letters that I hope will be helpful even when sending news of rejection. I have learned a tremendous amount, largely thanks to the fabulous reviewers who give up their time freely to provide detailed comments and suggestions to improve the clarity and quality of work the journal publishes. Now in my second term, I wanted to make a more strategic contribution to the journal. I set myself two objectives for this term of office. The first was to deliver a special issue on Developmental Language Disorder. This I did and on all counts it has exceeded my expectations. I am hugely proud of the content and our efforts to make it available to practising clinicians and educators.
My second goal was to introduce more explicit open science practices at the journal and on this score I have not fared so well. I have had this on the agenda for every editorial board meeting over the last two years. Each time I’ve gone away to get questions answered, gather more evidence, write author guidelines etc. Last summer I invited Chris Chambers to join our smaller meeting; he was incredibly helpful and I thought had alleviated any lingering concerns. Come the full editorial board meeting in February, I was certain we were going forward and I had a plan for delivery, including offering the option of registered reports. This would enable authors that choose this route to submit protocols which may be accepted in principle after peer review. However, the agreed actions from that meeting fell far short of what I was expecting and in my view are not workable.
I’ve actually been surprised about how upset I’ve been by this, especially considering that my own attempts to engage with open science practices have been fraught with obstacles. I hold the SCALES data – a population dataset including measures of language, behaviour, and cognitive development from school entry. We have screening data on over 7,000 children and detailed assessment data on over 500 children at two time points (and we are currently collecting wave 3 data). We are committed to making these datasets publically available, but have hit issues with the level of original consent (obtained 7 years ago) and the new GDPR requirements. Our first enquiry about registered reports was recently knocked back – although we won’t be finished with this phase of data collection until August, we had to start testing as soon as possible after funding was awarded, in order to see as many of the cohort as possible within the school year. We want to register our hypotheses and analysis plans, but of course the design/recruitment strategies were decided long ago and we can’t guarantee numbers as we are at the mercy of attrition. More concerning was the note that no longitudinal studies could be submitted as a registered report because previous waves of data would already be available (advocates of RRs – please let’s sort this out – large cohort studies with multiple variables at multiple time points are ripe for fishing expeditions!).
Despite these challenges, I remain convinced that this is the way forward. JCPP rejects the majority of submissions it receives (~80-85%). I’ve become increasingly demoralised by the papers I have to reject; with time you see the same sorts of mistakes repeated over and over again. For example, small sample sizes: if studies mention power at all, they are powered to detect large effects – something I rarely see in robust developmental psychology studies. This power issue is compounded by the fact that many studies reporting intervention trials have multiple outcome measures, half of which involve self/parent report and are therefore not blind to treatment status. Inevitably, only one or two of these will show significant change relative to a control group, and this is usually a marginal difference of small effect. This will be hailed as indicating great treatment promise, but I think such studies are very difficult to interpret and easily misrepresented. And so I end up rejecting them, outlining exactly why I cannot accept the paper. Increasingly I feel a pang of despair when I do – each of these studies represents a significant amount of funding, researcher time and energy, and investment by children, families and teaching staff. Over the years I’ve started to think that I would much rather offer advice at a point where it would help the design and execution of the study, not after the fact. After all, my ultimate goal is to improve the lives of young people with developmental disorder and we need robust and reliable research to do this. I’m under no illusion that registered reports will solve all of these challenges overnight, but I am convinced it is a step in the right direction. It is a step that signals we take this seriously, sets a stellar example of what “4*” research might be, increases the likelihood that studies are reported regardless of outcome, and facilitates future replication and meta-analysis.
The editorial board of JCPP is absolutely committed to publishing research of the highest quality. The person appointed to replace me is an outstanding scientist and I’m sure will be a huge asset to the journal and the field. I am hugely sorry to leave, but will now spend more time in my own lab, developing best practice, tackling some of the challenges we have faced, and training the next generation of developmental researchers to embrace open science. My time as an editor at JCPP, and my adventures learning more about open science, has fundamentally changed the way I think about research, and for that I am forever grateful.