Bias in research
I got a bit of flak from my last blog for suggesting that some highly experimental studies involving a small number of adults with a particular cognitive profile may not be relevant to the everyday lives of all people with autism. I was accused on Twitter and other places of being ‘biased’. At first I was a bit offended by this, but then I relaxed. One of the first things you learn as a psychology student is that everyone is biased. We can’t help it. It just seems to be part and parcel of the human condition. The second thing you learn though is that biases can be unhelpful and at worst, dangerous. In science biases can be particularly damaging, particularly if they cause scientists to ignore, or worse, suppress evidence that contradicts their biases, or prevents them from collecting or reporting such evidence in the first place (witness for example Susan Greenfield’s spectacular fall from academic grace). Scientists can’t completely eradicate biases, we are after all only human. The best we can do is to try and identify what our biases are, and take steps to minimize the impact of these biases on our scientific endeavours.
Thinking about bias brought to mind one of my favourite journal articles of recent years. The first thing I love about this paper is its wonderfully provocative title – the authors had the audacity to label me and people like me as WEIRD – white, educated, industrialized, rich and democratic. They then go on to make the observation that most psychology research, and more importantly the conclusions and theories about human perception and cognition derived from this research, is based almost entirely on WEIRD people, namely American (and British) university undergraduates. I can see why. I am deeply envious of my colleagues who can run a study on 50 undergrads in a matter of weeks, while it usually takes me a couple of years of travelling all over the place to have enough data to produce a publication! Nevertheless, the authors of this paper ask a simple question ‘are the data derived from an elite, 1% of the world’s population really relevant for the remaining 99% of human experience?’ And they make a convincing case that it is not. Guess what? Not everybody thinks, feels or processes information like a college student. In fact, when considered in relation to a more representative population sample (those in America or Britain who never made it to University or those from different cultures who never even went to a ‘school’ never mind university) these WEIRD people score so far from the population average that they could be considered ‘outliers’ and in some experimental contexts their data would be excluded. Does this mean we should abandon all research with undergraduates? Of course not, many groundbreaking theories have been derived from this subset of the population. Looking at the bigger picture also does not in any way diminish the strengths, talents or processing advantages of American and British undergraduates. It just means that if we want to draw any conclusions about ‘human’ cognition, we have to cast a wider net.
This has some bearing on the SCALES project. Why would I be crazy enough to take on a project of this size, which is a major logistical and intellectual challenge and frequently keeps me awake at night? Simple: I am interested in how language impairments are associated with impairments in other aspects of development (or ‘co-morbidity’). The literature suggests very strong associations: children identified with almost any kind of developmental disorder (ADHD, dyslexia, autism, Down syndrome etc.) are very likely to also have additional language impairments. Likewise, children with language impairments meet criteria for at least one other disorder at higher than expected rates (e.g. while dyslexia affects approximately 5% of the general population, some studies report that as many as 50% of children with ‘specific’ language impairment also meet criteria for dyslexia). The problem is that, with a few notable exceptions, the majority of studies reporting these associations are based on clinically referred samples of individuals, and these samples are inherently biased. In fact, this is so well-recognised that it even has a name, Berkson’s bias. The bias is that children who are having multiple development challenges (delayed language, clumsy, problems with attention and behaviour) are more likely to be noticed by parents and teachers and then referred for extra help from therapy or education services. However, there may be lots of children in mainstream classrooms who have whopping deficits in marking verb tenses (i.e. ‘he walk to school’ instead of ‘he walks or walked to school’) but are getting on just fine. This would not only be theoretically interesting (is language ‘special’ in the developing brain?), but practically important. If the ‘deficit’ we see doesn’t interfere with learning, making friends, or the child’s self-esteem, should we spend limited resources ‘fixing’ it? The only way to find out the extent to which language impairments are associated with other developmental concerns is to reduce the effects of Berkson’s bias, and the only way to do that is to recruit a more representative, population sample. That is what the SCALES project is trying to do.
I certainly have a few ideas about what I think we will find. I would prefer to call these ideas hypotheses, based on the extant literature and my own observations of children with language impairments over many years and across many clinical and educational settings. No doubt these hypotheses also reflect some my biases. I guess the test comes when we look at the data. If, in four years time, we find that my hypotheses are completely unsupported will I hang my head in shame, or try to hide the fact that I was wrong in case people think I am ignorant? No – I’d certainly be intrigued but also I think quite excited. Being wrong is often what moves science forward, challenges the dominant view and inspires new ideas and theories. How WEIRD is that? Well, based on the scientists I have known, I think not very. They may represent a biased sample, but I hope not.
Reference:
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 1-75.