Seeking Nirvana

“Nobody made a greater mistake than he who did nothing because he could do only a little.” – Edmund Burke


How do you know that smoking causes cancer?

Easy, right? scientists said so and they did lots of research to prove it. But what research did they actually do and how did they do it? If you’re anything like me, you probably have absolutely no idea.

In the 1950s two British doctors carried out a cohort study. This is when you look at a large group of people (40,000+ doctors in this case) over a period of time and study which conditions they suffer from and then try to match those conditions with other factors. For example those in the group getting lung cancer seemed to also overwhelmingly be the ones who smoked. Bingo, we have a correlation. 

I often wonder what would have happened if this were education research posted on twitter nowadays? My feeling is that as soon as it had been tweeted out countless blogs would have popped up to discredit it. 

Firstly someone would point out that correlation doesn’t always mean causation. Next we would read that doctors shouldn’t be trusted because ‘remember what happened with Thalidomide‘. Then, someone else would casually note that there must be hundreds of other factors which could influence these people, like diet and lifestyle. They would then pull out the classic educational trump card that ‘every smoker is different’ and that what affects one wouldn’t necessarily affect another. Next someone would ask for the authors to define exactly what they meant by ‘smoking’ are we talking pipes or roll ups? And just how many cigarettes makes one a smoker? Finally the coup-de-grace would be delivered with the comment that ‘my grandfather smoked 40 a day and lived till he was 100’.

Once the cloud of doubt was thick enough, everyone could go back smoking, safe in the knowledge that the imperfections in this research would protect them from cancer. 

The reasons used to dismiss research in education also exist in medical research and psychological research and somehow they seem to manage. 

Take a human beings for example. Each has their own unique genetic code. The differences are so extreme that some people can drink a little alcohol and suffer quite high levels of liver damage while others drink lots and are fine. Other can smoke their whole lives without getting lung cancer. Other people can die if given penicillin.  

Yes despite these differences when I buy a packet of painkillers it says “take one per day for adults” with no warnings about “unless you’re a middle-aged woman weighing between X and Y”. Somehow we can all just take one a day and ‘it works!’ But in education context is king and attempts to move the field forward can often be dismissed out of hand by this kind of low level niggling. 

The Nirvana fallacy is where ‘good’ is rejected because it isn’t ‘perfect’. It’s the enemy of ‘good enough’ or just ‘better than before’. And in education these kinds of improvements are exactly what we should be aiming for. There will never be a perfect method, but we should be asking are there ways of doing things that are a little better than how we’re doing them now. 

The Nirvana fallacy is not only apparent in criticisms of research, it also makes an appearance in two other areas of TEFL; textbooks and testing. Textbooks often don’t represent real language use, have contrived levels and use ‘old fashioned’ teaching methodology. They are often bland and designed by companies seeking to make a profit

None of this is controversial and there is plenty of research to back this up. But new textbooks come out all the time and are often better than the ones that precede them. Yet here again ‘better than before’ is not seen as good enough and instead there are many who seem to feel they should be thrown out altogether unless they are perfect. Of course ‘perfect’ here means applicable to every individual student’s needs regardless of the context, first language, learning preferences and cultural beliefs. They would also use the teaching methodology preferred by whichever teacher was using them and contain language appropriate and authentic for every knowable context. 

Tests too fall victim to the nirvana fallacy. In all areas of education it seems anti-test sentiment is high. Certainly tests can be powerful and life changing and bad tests are disastrous but again is that a reason to stop testing students or is it an argument for better tests? 

Testing is one of the most well-researched and evidence driven fields in education. The test ‘form‘ a person sits is the very tip of a complex and expensive test writing process which has been refined for decades. Tests also give us information on what a students is capable of, how well they’ve progressed and what they need to work on. Test writers and theorists go to incredible lengths to ensure tests are fair for students and yet I know of hardly any teachers who have positive views about testing. 

Bad research, bad textbooks and bad tests are all arguments for better research, better textbooks and better tests. It’s absolutely right that teachers should be critical of things that don’t work, and I will be there with them, pointing out sloppy research, crappy textbooks and poorly written tests. But should we dismiss the whole endeavour because it’s not perfect? Would we make similar arguments about other fields? charity for instance; ‘sure this well may supply clean drinking water but the hospitals are still in a terrible state and the government is unstable so why bother?’


We can still aim for improvements while admitting that things are not perfect. As Michael Long notes

The responsibility of professionals in any field is not to know the right answer, but to be able to defend recommendations in light of what is thought to be the right answer or the likeliest right answer (best practice), given what is known or thought to be known at the time. What is irresponsible is to throw up one’s hands and declare that no proposals should be made and defended until everything is known for sure (which will never happen). 




When nothing is better than something

 
 
Dr. Duncan MacDougall found that after humans die their weight changes by exactly 21 grams. He carried out his research on terminal patients and weighed them before and after death. He also carried out the same tests on dogs and found no weight change. No one can explain this strange phenomenon and the religiously minded as well as the New York Times wasted no time attributing it to the weight of the human soul.

An incredible and disturbing finding, were it true. 

Which it isn’t. 

 

While it’s completely true that what this experiment was carried out and that those were its findings, it’s equally true that it was a naff experiment. It’s easy to think any research is better than no research but bad research is often pretty useless; it tells us nothing and worse, sometimes it can even be dangerous. Dr. MacDdougall’s experiments were conducted on 6 people which is a horribly small number to warrant such extravagant claims. And that’s not all, quoting here from the blog “rationally speaking” the research had a number of other problems:

Not only was the experiment never repeated (by either MaDougall or anyone else), but his own notes (published in American Medicine in March 1907) show that of the six data points, two had to be discarded as “of no value”; two recorded a weight drop, followed by additional losses later on (was the soul leaving bit by bit?); one showed a reversal of the loss, then another loss (the soul couldn’t make up its mind, leaving, re-entering, then leaving for good); and only one case actually constitutes the basis of the legendary estimate of ¾ of an ounce. With data like these, it’s a miracle the paper got published in the first place.

Second, as was pointed out immediately by Dr. Augustus P. Clarke in a rebuttal also published in American Medicine, MacDougall failed to consider another obvious hypothesis: that the weight loss (assuming it was real) was due to evaporation caused by the sudden rise in body temperature that occurs when the blood circulation stops and the blood can no longer be air-cooled by the lungs. This also elegantly explains why the dogs showed no weight loss: as is well known, they cool themselves by panting, not sweating like humans do.

Ah, you may say, this was a long time ago before we had proper research. Well, while it’s true that scientific techniques improve all the time, research now, like research then is carried out by humans. So could a badly designed study get newspaper headlines these days? Over to Andrew Wakefield who ‘discovered’ a link between vaccinations and autism, research which led to, and continues to lead to parents not vaccinating their kids and thus the return of previously controlled diseases, such as mumps and measles as well as occasional deaths. Wakefield, who was eventually struck off the medical register, conducted his research on exactly 12 children so, twice as many as MacDougall’s study.  This didn’t stop theDaily Mail and other papers creating huge panic with this information.
 
Now small studies are not always problematic, but newspapers tend to have an undue influence on what people think and a story which might not get much (or any) attention in academia because of problems, such as its sample size, could have considerable influence if published in a newspaper. I wrote here about Memrise the amazing new technology which mean you can learn a new language in only 22 hours (disclaimer, for “language” read “some words” , for “22 hours” read “three months” and for “amazing new technology” read “flashcards and mnemonics”). Memrise is a good example of how the media can create excitement about something that really isn’t all that exciting
 
Recently there was a TEFL article in the guardian making the claim that the “argument was over, the facts were in” and that explicit grammar teaching was a must for EFL. Catherine Walker’s bold claims were marred by a couple of issues. Firstly the article wasn’t in a peer reviewed journal, it was in the Guardian (though even journals can get it wrong, and do, regularly, and spectacularly) and journalists are not experts and are therefore much more likely to let things slide that academics would probably pick up. Being a newspaper Walker didn’t have to provide any evidence for her claims, but when prodded by commentators listed the following:

Norris, J. M. & L. Ortega. 2000. Effectiveness of L2 instruction: a research synthesis and quantitative meta-analysis. Language Learning 50/3: 417-528.
Gass, S. & L. Selinker. 2008. Second Language Acquisition: an Introductory Course (Third Edition). New York: Routledge/Taylor.
Spada, N. & Y. Tomita. 2010. Interactions between type of instruction and type of
language feature: a meta-analysis. Language Learning 60/2: 1-46.
Spada, N. & P. M. Lightbown. 2008. Form-focused instruction: isolated or integrated?
TESOL Quarterly 42: 181-207.

To be fair to Walker, meta-analysis are the creme de la creme of research and are positioned at the very top of of the hierarchy of evidence pyramid. However there are still some problems with the piece. The first flaw is that the headline for the article is misleading. Articles from 2010, 2008 and 2000 can’t be run in 2012 with the headline “the evidence is finally in”, without stretching the “finally” beyond recognition. Secondly the headline makes the claim that evidence shows that grammar teaching is effective, yet later in the article this is watered down to:

 



However, evidence trumps argument, and the evidence is now in. Rigorously conducted meta-analyses of a wide range of studies have shown that, within a generally communicative approach, explicit teaching of grammar rules leads to better learning and to unconscious knowledge, and this knowledge lasts over time.

That is, if you are using the Communicative Approach, it is better to have grammar taught explicitly. So this is not so much a debate about the value of grammar teaching but a debate about the value of teaching grammar explicitly within a certain method. The title of the article may have had us all rushing back to the oft mocked (but pretty widely used) Grrammar Translation method.  Another possible problem is the conflict of interests. Walker has a written a number of grammar textbooks and while this doesn’t mean she’s wrong, the possibility of bias is there; “shock! grammar teacher claims teaching grammar works!”. 
 
So language articles are often annoying because they get tweeted and retweeted when the findings may be problematic or in some cases nonexistent, like a story doing the rounds  at the moment. Apparently, English is not a Germanic language but a Scandinavian one. For years linguistics have been wrong and this new research shows conclusively that English comes from Scandinavian not from Old English. Except it doesn’t because, as far as I can discover, there is no research. Yes, there are researchers and yes there are news articles and yes there is even some evidence of conference talks but I can’t seem to find a paper published in a major peer-reviewed journal (please someone link to it if you can find it).
Now, I’m out of my depth with the argument as to whether English is or isn’t a Scandinavian language perhaps it really is but what I can say is that a massive claim like this, if correct, would make the careers of both the authors. I can also safely state that a claim which has such large potential, needs a equally large amount of evidence to back it up. To use the Sagan Standard “Extraordinary claims require extraordinary evidence”. Judging by the comments on some of the blogs that have reported this, I’m going to tentatively suggest that that evidence will not be forthcoming.
  
The king of getting media attention with little research is of course, Chomsky. Slayer of the evil behaviorists, discoverer of the mysterious UG, Noam wins the prize by virtue of having done exactly 0 research to test his theories. His ideas, which have held sway over linguistics for 60 years, were thought up by him, and then left for others to argue about.  People who actually took the trouble to look into and test Chomsky’s claims found him to be wrong, wrong and more wrong.*  Geoffry Sampson writes:

Hang on a minute,’ I hear the reader say. ‘You seem to be telling us that this man [Chomsky] who is by common consent the world’s leading living intellectual, according to Cambridge University a second Plato, is basing his radical reassessment of human nature largely on the claim that a certain thing never happens; he tells us that it strains his credulity to think that this might happen, but he has never looked, and people who have looked find that it happens a lot.’

Yes, that’s about the size of it. Funny old world, isn’t it! (2005:47)


References

Sampson, Geoffrey. 2005. The Language Instinct Debate. London & New York: Continuum

*I’m really really out of my depth on Chomsky but if someone wants to come and put me right I’ll be happy to listen.

NLP notes

I just saw this post get retweet. I had to quickly knock out a reply to it here. It deals with one of my pet hates, Neurolinguistic Programming which has become more popular over recent years in the EFL world but which makes some quite remarkable claims. I’m currently trying to a get an article on NLP published and I don’t want to repeat too much of that here, but  since it was retweeted by someone quite influential I thought I would dash off this response. I wrote this in about 20 mins so sorry about typos etc.
 
NLP is a weird therapy type of system which was dreamt up in the 1970s and makes spectacular claims about both the human body and what NLP itself is capable of. Similar in genre to books promise to help you get rich in 7 easy steps or to eat yourself thin, NLP makes some quite spectacular claims. One book (Agnes 2008:3) for example claims that using NLP can help you to:
 
Be what you want to be
 
Have what you want to have
 
Do what you want to do
 
Have the personal success you want now
 
Be more aware of your thoughts
 
And who wouldn’t want all that. Yet there is actually no research that supports any of the claims that NLP makes. This is hardly surprising when you realise just how odd those claims are. NLP practitioners, like the author of the blog that was retweet claim that watching a person’s eye movements can tell you what kind of learner they are. That is, in the dubious VAKOG sense of learner styles
 
You can also listen to the pitch of someone’s voice or check the way they walk to find out what kind of learner they are. If this doesn’t work then check out the words they use. A person who says “I see what you mean” is visual and and someone who tends to say “I get your drift” is probably kinaesthetic. These are called predicates in the NLP world and no, I’m not kidding, -this is really what they teach.
 
If you check the blog you see this quote: 



For me, one of the most important core concepts of NLP is the recognition of differences in cognitive style – or what NLP calls “representational systems”. There are five of these systems (Visual, Auditory, Kinaesthetic, Olfactory and Gustatory) of which the first three are most commonly used

 
Isn’t it funny how the olfactory and gustatory learners are always left out? No one wants to deal with learners who learn through smelling and tasting? too silly even for NLP perhaps? Also, if we’re going to use this old-fashioned five sense model then what happened to touch? No one fancy teaching touchy learners?
 
The blog mentions Revell and Norman’s book on NLP in EFL which I would advise you to take a look at. There are some quite impressive claims in there, even “live longer with this 3 minute exercise” -worth the price of the book alone I would have thought. The blog also quotes some NLP sayings such as: 


 

but there is no failure in NLP, only feedback 

This sounds great and appeals to me as a teacher but isn’t it just word play? The Chinese girl who paid out £2,500 for a pre-sessional course, not making the grade and being sent home probably wouldn’t see that as ‘feedback’. Some other claims that NLPers make are these:
There is no failure in learners only in the teacher’s intervention (Millrood 2004:29)

There is no such thing as reluctant learners only inflexible teachers (winch 2005:np)

All behaviour has positive intentions (Revell and norman 1997: 106) 

Now it might just be me but these claims are seriously questionable. Learners can fail, they can be forced to study English and they can almost certainly act with negative intentions.

The author of the blog also claims “I’m no NLP expert but…”. Well, don’t worry, that can soon be remedied. It’s easy to become a “master practitioner” of NLP in one short 10 day course. It will only set you back a few thousand pounds and it’s so simple that even a cat could do it

The fact that so many teachers have bought into this dubious and expensive practice doesn’t really bother me, that’s up to them. However the fact that they are wasting students’ time (and money) by staring into their eyes, or listening to which words they use does. You can find examples of teachers doing this kind of things here, here and here. Of course these teachers think these things “work” and that’s great, -but just remember, the students didn’t sign up for pseudoscience, they signed up to learn English.

 
 
references
 
Agness, L. 2008. Change your Life with NLP Edinburgh: Pearson education LTD
 
BBC. 2009. Cat Registered as Hypnotherapist. In BBC news. Retrieved September 20 2012, http://news.bbc.co.uk/1/hi/8303126.stm
Millrood, R. 2004. ‘The role of NLP in teachers’ classroom discourse.’ ELT Journal. 58(1): 28-37
Revell, J. and Norman, S. 1997a.  In Your Hands – NLP in ELT. London: Saffire Press.
Winch, S. 2005. ‘From Frustration to Satisfaction: Using NLP to Improve Self-Expression.’ in EA Education Conference, English Australia, Mercure Hotel, Brisbane, QLD.