Seeking Nirvana

“Nobody made a greater mistake than he who did nothing because he could do only a little.” – Edmund Burke


How do you know that smoking causes cancer?

Easy, right? scientists said so and they did lots of research to prove it. But what research did they actually do and how did they do it? If you’re anything like me, you probably have absolutely no idea.

In the 1950s two British doctors carried out a cohort study. This is when you look at a large group of people (40,000+ doctors in this case) over a period of time and study which conditions they suffer from and then try to match those conditions with other factors. For example those in the group getting lung cancer seemed to also overwhelmingly be the ones who smoked. Bingo, we have a correlation. 

I often wonder what would have happened if this were education research posted on twitter nowadays? My feeling is that as soon as it had been tweeted out countless blogs would have popped up to discredit it. 

Firstly someone would point out that correlation doesn’t always mean causation. Next we would read that doctors shouldn’t be trusted because ‘remember what happened with Thalidomide‘. Then, someone else would casually note that there must be hundreds of other factors which could influence these people, like diet and lifestyle. They would then pull out the classic educational trump card that ‘every smoker is different’ and that what affects one wouldn’t necessarily affect another. Next someone would ask for the authors to define exactly what they meant by ‘smoking’ are we talking pipes or roll ups? And just how many cigarettes makes one a smoker? Finally the coup-de-grace would be delivered with the comment that ‘my grandfather smoked 40 a day and lived till he was 100’.

Once the cloud of doubt was thick enough, everyone could go back smoking, safe in the knowledge that the imperfections in this research would protect them from cancer. 

The reasons used to dismiss research in education also exist in medical research and psychological research and somehow they seem to manage. 

Take a human beings for example. Each has their own unique genetic code. The differences are so extreme that some people can drink a little alcohol and suffer quite high levels of liver damage while others drink lots and are fine. Other can smoke their whole lives without getting lung cancer. Other people can die if given penicillin.  

Yes despite these differences when I buy a packet of painkillers it says “take one per day for adults” with no warnings about “unless you’re a middle-aged woman weighing between X and Y”. Somehow we can all just take one a day and ‘it works!’ But in education context is king and attempts to move the field forward can often be dismissed out of hand by this kind of low level niggling. 

The Nirvana fallacy is where ‘good’ is rejected because it isn’t ‘perfect’. It’s the enemy of ‘good enough’ or just ‘better than before’. And in education these kinds of improvements are exactly what we should be aiming for. There will never be a perfect method, but we should be asking are there ways of doing things that are a little better than how we’re doing them now. 

The Nirvana fallacy is not only apparent in criticisms of research, it also makes an appearance in two other areas of TEFL; textbooks and testing. Textbooks often don’t represent real language use, have contrived levels and use ‘old fashioned’ teaching methodology. They are often bland and designed by companies seeking to make a profit

None of this is controversial and there is plenty of research to back this up. But new textbooks come out all the time and are often better than the ones that precede them. Yet here again ‘better than before’ is not seen as good enough and instead there are many who seem to feel they should be thrown out altogether unless they are perfect. Of course ‘perfect’ here means applicable to every individual student’s needs regardless of the context, first language, learning preferences and cultural beliefs. They would also use the teaching methodology preferred by whichever teacher was using them and contain language appropriate and authentic for every knowable context. 

Tests too fall victim to the nirvana fallacy. In all areas of education it seems anti-test sentiment is high. Certainly tests can be powerful and life changing and bad tests are disastrous but again is that a reason to stop testing students or is it an argument for better tests? 

Testing is one of the most well-researched and evidence driven fields in education. The test ‘form‘ a person sits is the very tip of a complex and expensive test writing process which has been refined for decades. Tests also give us information on what a students is capable of, how well they’ve progressed and what they need to work on. Test writers and theorists go to incredible lengths to ensure tests are fair for students and yet I know of hardly any teachers who have positive views about testing. 

Bad research, bad textbooks and bad tests are all arguments for better research, better textbooks and better tests. It’s absolutely right that teachers should be critical of things that don’t work, and I will be there with them, pointing out sloppy research, crappy textbooks and poorly written tests. But should we dismiss the whole endeavour because it’s not perfect? Would we make similar arguments about other fields? charity for instance; ‘sure this well may supply clean drinking water but the hospitals are still in a terrible state and the government is unstable so why bother?’


We can still aim for improvements while admitting that things are not perfect. As Michael Long notes

The responsibility of professionals in any field is not to know the right answer, but to be able to defend recommendations in light of what is thought to be the right answer or the likeliest right answer (best practice), given what is known or thought to be known at the time. What is irresponsible is to throw up one’s hands and declare that no proposals should be made and defended until everything is known for sure (which will never happen). 




Why we need evidence part 2: How we know what isn’t so

Are you more likely to die from a car accident or from cancer? And just how much more likely is the one than the other

While writing this post, the excellent ‘EFL teachers shouldn’t prefer blonds’ appeared. It is definitely worth a read and gets right at the heart of what I’ll talk about here, even mentioning the availability error.
 

The availability error/heuristic means that people tend to favour information that is more available to them.  For example, If you ask people whether they are more likely to die from a car accident or cancer many people would suggest that they were perhaps about equal, after all cars are very dangerous. Of course, people dying of cancer doesn’t make the news very often and so the fact that you are over 10 times more likely to die of cancer than being in a car accident, isn’t as ‘available’. Similarly, people may worry about terrorists or shark attacks, but the likelyhood or either of these happening to you is so remote as to be almost meaningless. That said, both of these are still more likely than winning the lottery, but in the UK round 2 million play it every week.  

As Andrew Walkley points out, the same goes for teachers (and material writers) understanding of language. We think we know what people say, we think we know how people talk but actually we often have no idea. We think we know which words are ‘common’ and which are less so but as this game shows, we probably get it wrong as often as we get it right (thanks to Rachel for the link). This is because humans don’t do statistics very well.  

For example, imagine if I told you I was going to spend a lesson teaching students about the the word ‘just’. you may be horrified at my wasting so much time on a word like that. However examining Nottingham’s corpus of spoken English reveals that “just” is the 31st most commonly used word in Spoken English (O’Keffee et al 2007: 35). Other words are “yeah” (no.12), “mm” (37) and “er” (38).

Generally our intuition about language is wrong. What we end up teaching students is what Lewis calls “TEFLese” (2001: vii) and Stubbs notes that many learners may be able to ‘speak grammatically, yet not sound native like’ (2006:115) As long ago as 1998 this approach was being questioned:
 

Working on educated guesswork or hunches when writing dialogues and transactions for coursebooks or when selecting language to teach is highly questionable. (williams 1998:53)
 

For example, maybe today you taught a class.  And depending on where you work (China, Korea and Japan, yes, I’m looking at you!) perhaps the class featured a dialogue like the following:

 
A: Hi, How are you?
B: I’m fine thank you, and you?
A: I’m fine too, thank you.
 
Now it’s easy to pick fault with this for being unnatural, but what about this one (Nunan and Lockwood in Carter et al 2005:79):
Patient: Could I make an appointment to see the doctor please?
Receptionist: certainly, who do you usually see?

Patient: Dr Cullen
Receptionist: I’m sorry but Dr Cullen has got patients all day. would Dr. Maley do?
Patient: Sure
Receptionist: OK then. When would you like to come?

Patient: Could I come at four o’clock?
Receptionist: four o’clock? Fine. Could I have your name please?

If you’re like me then you have probably taught this kind of thing before. It looks pretty authentic, doesn’t it? Certainly better than the 1st example. Yet reality looks quite a bit different (Burns at al 1996 in Carter et al 2005:80):

 
Receptionist: Sorry to keep you waiting.
Patient: That’s alright um I’m just calling to confirm an appointment with Dr X for the first of October
Receptionist: oh…
Patient: Because it was so far in advance I was told to.

Receptionist: I see what you mean, to see if she’s going to be in that day.
Patient: That’s right
Receptionist: Oh we may not know yet.
Patient: Oh I see.
Receptionist: First of October…Edith….yes.

Patient: Yes
Receptionist: There she is OK you made one. What’s your name?
Patient: At nine fift…
Receptionist: Got it got it.
 

Now whether or not we expose students to this kind of dialogue is an argument I’m not going to get into here, but the point is that despite what we think we know, we are often very bad at judging what actually is.

And if we have no idea about how language is really used how can we expect to make the best use of our students’ time? This is where the evidence discovered by corpus linguistics is invaluable to teachers. Computers don’t suffer from the availability error. Corpus has revolutionised dictionaries, textbooks and language teaching. But it has only managed this because we were so wrong, so often. If our intuitions about language had been largely correct, it’s doubtful corpus would have made the impression that it has.


References

Carter, R. Hughes, R and McCarthy, M (2005) telling tales: The spoken language and materials development in Tomlinson, B (Ed.), Materials development in language teaching. Cambridge UniversityPress 67-88.

Lewis, M. (1993). The Lexical Approach. Hove:Language Teaching Publications.

Williams, M. (1998) Language Taught for Meetings and Language Used in Meetings: Is there Anything in Common Applied Linguistics 9/1: 45-58

Stubbs, M. (2006) language Corpora. In Davies, A. & Elder, C. (2006) (eds.) The Handbook of Applied Linguistics. Malden M.A. Blackwell. 106-132