At the end of last year Masatoshi Sato kindly agreed to sit down with me and discuss Evidence Based practice in ELT. Prof Sato is a advocate of research being used to support teaching. He has written books on evidence based pedagogy and has carried out work on teachers’ opinions of research and how it can support teaching.
Here we talk about what Evidence Based approaches mean to him, the gap between teachers and researchers and some of the problems with research.
So good riddance to 2020! A year in which teachers everywhere were forced to become tech experts. I’m curious how well those talks on the wonders of tech will go down in future conferences now we have all experienced the brave new world of Zoom.
This year has also shown us the power of science at work. When the 1914 pandemic happened, there was no vaccine and 50 million people eventually died of the disease. In contrast, after 1 year there are a number of vaccines for COVID19 all being rolled out at the same time. This is because science is cumulative. We can build on what we learn in order to do things better in the future. In ELT, research is still at a very early stage. It’s tempting to make the claim that research is useless and can’t tell us anything about the classroom (and many people do). I think this is shortsighted. There are advances we can make and these advances will allow us to make other advances in the future.
Criticism can be hard to take. In the book “kindly Inquisitors” the author relays the story of Georg Cantor a mathematician who “lost his mind because of the hatred and animosity against him and his ideas by his teacher Leopold Kronecker: He was confined to a mental hospital for many years at the end of his life” (2014: 296) Kubota suggests that “The field of L2 education by nature attracts professionals who are willing to work with people across racial boundaries, and thus it is considered to be a ‘nice’ field”. But is TEFL too nice?
It might seem an odd question to ask, after all, how can people be too nice? For me, sometimes the desire to protect relations and be kind tips over into a kind of censorship. This happens when criticism is withheld or watered down to protect people’s feelings. Here are some personal anecdotes:
Before I gave a talk once, the organisers asked me to remove references to certain people in a talk they had invited me to give.
While writing an piece I was asked if I could remove references to authors who held the views I was criticising.
Maybe I was wrong in these situations, after all, it is entirely possible to criticise a position without saying who holds it. That said, when discussing the prevalence of neuromyths in ELT, for instance, might it not be important to note that prominent figures are promoting these things? Or is being discreet more important?
Personally I find it a little frustrating when reading an article that says something to the effect of “many believe that correction is not useful” or “some people disagree with this idea” and not seeing a link to who it is who is making these points. I would like to go away and read their work and see exactly what they say, but instead I just have to take the writer’s word for it.
Should we avoid giving this useful information for fear of offending? I tend to agree with Rauch who notes, “people who are hurt by words are morally entitled to nothing whatsoever by way of compensation. What is the right answer to the person who demands something because he is offended? Just this: ‘Too bad but you’ll live'”(80)
So am I for people saying whatever they want? Well, not quite…
One of my ELT heroes is Mike Swan. His articles seem to cut through nonsense and provide a clear and fair examination of authors. At the same time he didn’t pull his punches and wasn’t afraid of frank criticism. However, if you’ve seen his collected works you’ll notice that the articles are prefaced with his current reflections on them. Before one of his most scathing articles he writes the following:
‘The tone of the articles was consequently excessively polemic, anti-academic, and at times downright rude. I now offer my belated apologies to the several distinguished scholars for whom I showed less respect for me certainly deserved.’ (2013:1)
Reading the essay that Swan is apologising for, I was struck by how mild it is compared to blog posts and tweets we can see daily. Swan, I suppose, belated recognised that criticism is hard enough to take without adding unnecessary venom.
Another negative side effect of overly unpleasant criticism is that your critics can dismiss you very easily. “I don’t object to what you said, just the way you said it. Let’s discuss that instead.” This kind of tone policing has been examined by Andrew Old, who notes that the subjective nature of ‘tone’ means that “almost any style of disagreement can be objected to on this basis”. Notably though, he draws the line at insults, threats and rudeness and is quick to block those he feels crosses the that line.
An academic issue
Academic writing is often impenetrable, vague and dull. Language seem to be used at times with the purpose of confusing rather than elucidating. However, this dullness and cold objectivity can perhaps be used to temper the anger we might feel when reading a piece criticising something we have written.
Writing that someone’s view “does not seem to be supported by the evidence” might be easier to take than describing someone’s views as ‘crap’. Saying someone’s opinions are ‘moronic prattling’, while technically not insulting the person, is unlikely to lead to a productive debate. In fact, it is probably much more likely the other party will entrench their position rather than come round to your way of thinking. Sure, it’s great to get patted on the back after DESTROYING someone with FACTS and LOGIC but how much does this kind of rhetoric effect any actual change?
It can also create a rather toxic environment. If the discussion becomes increasingly extreme, only the extremeophiles thrive. Others will choose not to engage. We thus lose all but one type of voice and it becomes an intellectual cul-de-sac of sorts.
The importance of criticism
“Controversy is good, it makes us think” Scott Thornbury, IATEFL 2016 Plenary
Is it really a kindness to not criticise idea because you want to ‘nice’? Is it a good situation if everyone disagrees with you but is cowed into silence or should we encourage people to say what they think? What if what they think will effectively silence others? It’s not an easy question to answer. I don’t know where the lines should be drawn and I don’t think people should have to put up with people being abusive to them. What I can say is that I think I’ve learnt far more from criticism than praise, no matter how hard it is to hear. To quote Kindly Inquisitors again “a no offence society is a no-knowledge society” (2014: 297).
In SLA research, one finding seems beyond reproach is that there is a set order in which students acquire grammar. This “internal syllabus” cannot be overridden and thus textbooks that present grammar unit by unit are pointless and worse ‘unnatural’, because students are unable to learn what is taught until they are developmentally ready.
Although this research is often discussed, I have found the details are rarely forthcoming. I was curious to know two things about these landmark studies. Firstly, how was ‘acquisition’ measured, and secondly what do they consider to be ‘grammar’? In this post I will be looking at the morpheme studies.
1. what falls under ‘grammar’ and what does not?
In the morpheme studies, a set of roughly 10 morphemes are usually researched. These vary slightly such as when researchers separate articles into ‘the’ and ‘a’, or look at long and short plural sounds but in general they don’t differ much between researchers. The list of morphemes include such things as plural forms (dogs), Copula (is) (He is happy), auxiliary be (he’s coming), irregular past, regular past, articles and possessive -s (John’s cat).
There is a lot that teachers would consider ‘grammar’ that is not included. For instance:
I should play tennis. (Modals)
If you like it, then buy it. (Conditionals)
I’ve told you already (perfect forms)
What are you doing? (Questions)
2. How is acquisition measured?
In the morpheme studies, a test subject is said to have acquired a grammatical form if they can produce it correctly in a test. The test, called a Bilingual Syntax Measure (BSM), is usually carried out on children and involves showing cartoon like pictures and eliciting language from the subjects. A researcher will, for instance, say ‘here is a girl, now there are two of them. So there are two _____?’ this is known as an ‘obligatory context’ as students have to use the correct form to answer.
The next stage is that researchers score the learners depending on whether they produce the correct form or not. For instance (Dulay and Burt 1974):
totally correct ie. “she’s dancing” (2 points)
half right, ie. “She’s dances” (1 point)
wrong ie. “She’s dance” (0 points)
The scores of the entire group are then added up and plotted on a chart. The equation used was the sum of the whole group / the number of possible points x 100. If more learners correctly produce plural -s than produce possessive -s, then the researchers claim that plural -s is acquired before possessive -s.* In the morpheme studies a form was said to be ‘acquired’ if subjects produced it accurately when elicited 90% of the time.
What did they find?
Researchers seemed to find that all students acquired language in the ‘roughly’ the same order regardless of their L1. For instance Mitchell and Myles (2004: 43) argue that these results suggest ‘second language learners are guided by internal principles that are largely independent of their first language’.
The interesting thing to notice when looking at this table is that the orders found were not actually the same between researchers, which is a little surprising for a ‘universal’ order. That said some researchers seemed not to mind and grouped the morphemes into ‘sets’ which are acquired in order.
The eagle-eyed among you will perhaps spot that there are still some outliers here such as “articles” appearing in stage 2 yet 1st in Dulay and Burt and 11th in Hakuta.
Issues with this research
I was interested to discover, that despite the ‘Holy Grail of SLA research‘ status that the morpheme studies have achieved, they have been under scrutiny for almost as long as the have been around. Some of the criticisms levelled at this research is as follows (apologies for not being able to properly source the origin of these).
morphemes with different meanings (a/the) were grouped together in some studies
What was classed as ‘grammar’ was a very limited number of morphemes
most of the early research was carried out on ESL learners, not EFL students
The orders vary in different papers, notably Hakuta 1974 (n-1)
students were all grouped together to obtain results, hiding individuals or national groups who may not have followed the “natural” order.
the studies did not look at acquisition over time but rather just took a snapshot
accuracy order does not necessarily mean acquisition order
Students’ overuse of the target morphemes was not counted
The “universal order” is more accurately thought of as the “Spanish student order”
(Note some of these criticism have merit and others less. Check Larsen-Freeman and Long 1991 for a more detailed explanation.)
Notably, the claim that the order is universal has started to look suspect as L1 does indeed seem to have some influence on L2 (something that will not surprise most teachers). Luk & Shirai (2009) have argued that researchers continue to promote the order as ‘universal’ ignoring the evidence that it seems to be affected by a students’ L1. Corpus research, for instance shows that students seem to acquire morphemes in a different order. For instance, Japanese has a possessive particle ‘の’ but no plural particle and Japanese students seem to learn possessive -s before they learn plural -S (Anecdotally, this chimes with my experience). Hakuta’s study had a similar results and interestingly, Hakuta found that articles, which do not exist in Japanese, were late acquired by the Japanese student he studied.
So the ‘holy Grail’ seems to have a few cracks in it. One author who believes that the morpheme studies have been used to make claims that they could not support is Mike Swan, who notes:
We have no reason at all to believe that the learning of most grammatical items is constrained in this way: that for yet-to-be uncovered developmental reasons, students might need to learn comparatives before relative pronouns, dativizing verbs before quantifiers or infinitives of purpose before possessive ’s. To claim that learnability findings preclude the operation of a grammatical syllabus is a large and unjustified leap across a wide logical gap.
Why do some people continue to believe that climate change is not a threat despite the overwhelming scientific evidence that it is?
One theory is that the people who reject the evidence for climate change are doing so because they are uninformed. The idea is that if they were only a bit more educated on science they would suddenly realise the error of their ways. This is called the knowledge deficit model.
Researchers have sought to investigate how accurate this model is. They tested the hypothesis that a lack of knowledge about climate change predicted thinking it was less of a risk. They tested subjects scientific literacy and compared that with views on climate change.
If their hypothesis were true we would expect to see that risk perception would rise with more knowledge. It would look a bit like this.
In fact they found this.
There was no correlation between scientific literacy (as measured by a test) and belief that climate change was a risk. Some of those who believed climate change was a big problem knew a lot about science and conversely some who knew little about science thought the risk was great. In short, the knowledge deficit model was a bust.
However, when the researcher compared political affiliation with belief in climate change they found the correlation they were looking for. It seems that being left wing was a much bigger indicator of believing climate change was a risk and vice versa for being right wing and this was regardless of scientific literacy.
The researchers propose that when a subject, like climate change, becomes politically partisan, facts will do very little to alter opinions. David McRaney has dubbed this “tribal psychology“. When something becomes a matter of identity for your in-group, then the evidence ceases to matter very much.
Take something as simple as wearing face masks during a pandemic. A year ago it would have been hard to image people physically attacking each other over an issue like this, but here we are. McRany has an excellent Podcast on this subject. He notes that masks may now be a badge of group loyalty, like putting pronouns in your bio. These findings are especially interesting when thinking about research and teacher beliefs.
In education most people are already left wing, yet the battle between trad and prog teacher “tribes” rages on with almost daily outrages during which members can prove their group loyalty. Today, it’s language use in schools, before that it was exclusion and before that it was something else. Topics change, but the dance remains the same.
It’s very “natural” for people to pick a side and to stick to that side regardless. As McRaney notes, our evolutionary history as social animals means that group membership could have meant the literal difference between life and death. It’s also a very frustrating part of human psychology leading to irrationality, partisanship, class divisions, religious intolerance, and so on.
A large body of psychological research shows how when we form in and out groups we start to view those two sides very differently. Our side is hard done to, sensible, upstanding and decent. The other side is getting away with murder, idiotic, immoral and craven. If we make a mistake, it was an accident, if they do it was calculated. worst of all, McRaney notes that people would rather be wrong than out of good standing with their “tribe”.
If people’s identities are wrapped up in a certain sets of beliefs and if those beliefs form an in-group/out-group dynamic it is likely that everything will be viewed through that lens. it is also likely that, in this situation, no amount of evidence will change minds. A good example of this is phonics research, the results of which have been resisted for decades.
So here are some suggestions to try to avoid our natural tendencies towards “tribal psychology”. This advice is aimed mostly at myself, but if you find it useful, then great. I suggest these in the full knowledge I will probably be unlikely to follow them very well.
Try not to become too invested in any one side
This is very difficult, but the more we see the other side as the enemy, the easier it is to start seeing everything through a partisan lens. Next, we may stop looking at the evidence all together and just side with our ‘tribe’.
Try to find things you can agree with
Whenever possible and no matter how small it is, try to find points of agreement between you and people in your out group.
Try to stay agnostic where possible
We would like the research to say “X” but it doesn’t exactly say “X”. Well, the best thing we can do is just not have an opinion. “I don’t know” is a valid opinion.
Be dubious of research that supports your world view
Paper finds thing I want to be true is wrong *scoff* “what was the sample size?” “Let me check the methodology section!” “This author is right winger!”
Paper finds thing I want to be true is true! RETWEET without reading! “OK the sample size isn’t ideal but it’s a promising piece of research!” “who cares what the author’s politics are?!”
Make appeals to people on their terms
Learning styles advocates never seem to get particularly fazed by the charge that the practice lacks evidence. They do, however, get upset by the idea that it pigeon holes students. My guess is learning styles advocates like the theory because it represents individualism and student centric learning. Questioning that claim resonates more than pointing out the lack of evidence.
Facts aren’t enough
Research tell us that the knowledge deficit model is flawed. If we can’t get people to believe in the threat of climate change despite the evidence, there is little hope of convincing them about anything else. A research based approach has to consider not only the facts that can be gleaned from research, but people’s feelings too. Bashing teachers over the head with research is unlikely to change minds but it may cause them to resent you. It’s then fairly easy for teaching experts to dismiss research as “irrelevant for teachers” and find a receptive audience.
Here are a random collection of things which I think are really important but which are not really evidence based.
5. Deal with behavioural issues as soon as possible
If you’re like me, then having a student on their phone or basically doing something distracting will put you off your stride. I think most of the time, with adult learners there’s no malice, they are just thinking about their own problems.
If something is bothering you in class then deal with it as soon as possible. If a student is bothering you, then it’s likely they are bothering other students in the class. They suffer and you suffer. I usually ask to see anyone who is doing something they shouldn’t after the class. I don’t think this has to be a difficult conversation, just “don’t do X”. Setting clear rules on day 1 is a big help.
4. use coloured paper with wide margins and light ink when making cards
It might seem a bit “TEFL” for an EAP teacher but it really saves time. If you are producing cut-ups then pre-make a document which has side margins wide enough that you don’t have cut them at all. Make the spaces between the sentences as large as possible because there is nothing worse than chopping a few words off one set. Finally, print each set on different coloured paper to make them easy to sort.
3. Everything usually takes longer than you think
I have a compulsive fear of running out of materials before the end of the lesson and I know that I am not alone. I did have a tendency to over plan for a long time and end up not getting through the material. I’m better at this now, and there are some easy ways to control the time in the lesson. These usually relate to how you give the answers (just write them all up and let the students check themselves or nominate people and have others confirm the answers). You can also, – shock horror – do activities twice if you find you have too much time. It might seem lazy, but the students will probably benefit from the practice.
(I only actually ran out of material twice in 20 years, which isn’t bad.)
2. Learn student names
I’m not great with names but I make sure that I always do this in the first class. It’s really cringy by the third lesson to be pointing and saying “you” and I think it makes the classes less effective. If I can’t learn them because there are too many students, or because I will only see the group a few times then I give them a name plate of some kind.
Do the damn worksheet
This is a cardinal rule for me and yet the one that I observe the least, much to my detriment. Do the damn worksheet yourself. It’s only once you are in class that you realise that Q3 isn’t actually possible in passive, or that you can’t explain something about the grammar point you are supposed to be focusing on, or you don’t actually know why “pain” is countable in “what a pain”, or how to explain the word “innovative”, or why “try” takes infinitives and gerunds while”trying” only takes infinitives, and on and on for 20 years. It is only when you try to do the worksheet that you realise the planned activity is not physically possible or that it would take 2 hours to complete. Do the damn worksheet!
I am currently reading the new book, Science Fictions by Stuart Ritchie. It deals with meta research (research about research) and outlines all the ways in which science is currently going wrong. In one section dealing with retraction, Ritchie notes that “1.97 per cent of scientists admit to faking their data at least once” and suggests that that number is probably an underestimate as people are unwilling to admit to things like this even when asked in anonymous surveys.
This number means that for every 50 papers published in ELT one is likely to contain faked data. Some of these cases have come to light in biology and psychology and this led me to wonder if there were many retractions in ELT. So I asked twitter.
Marc Jones instantly found two (here and here) both of which were retracted due to plagiarism. One was plagiarism of another scholars work and the other was self plagiarism (submitting the same paper for double credit). I also found one from The journal of Computer Assisted Language Learning retracted for (self) plagiarism. So far no fraud.
The journals Applied Linguistics, ELTJ and TESOL Quarterly have apparently had 0 retractions to date. The database is not complete though and I was also sent this piece which was retracted from the prestigious “language learning” journal. It’s not that clear what went on here but it seems like one author noticed errors in the data and requested a retraction.
As an interesting aside, Richie notes that people continue to cite retracted articles even after they have been retracted. I was curious about this so I used google scholar to check citations of the 2003 Language Learning article. There were hits from 2018 and even from 2019. I don’t know the date of the retraction but I feel pretty confident it was well before these dates.
Out of thousands of papers in ELT I could find only two that were withdrawn due to data issues. So either ELT is a paragon of virtue or we haven’t got very good at sniffing out fraud yet.
But how easy is it to get a journal to retract a paper? While researching an article on learning styles, I came across a couple of very curious articles. The first was Hyland 1993, the second was Hyland 1994. While reading the 1994 article I got a strange sense of Deja Vu:
In essence, learning style research suggests that people make sense of the world in different ways and these ways are partly created by cultural experiences (Hyland 1994)
Learning style research therefore suggests that people make sense of the world in different ways, more importantly however, these ways are partly created by cultural experiences (Hyland 1993)
Eight Japanese universities participated in the survey with 265 undergraduate students responding. The questionnaire was also administered to 140 Japanese students at various English proficiency levels at a tertiary college in New Zealand to determine whether overseas study influenced modality and group preferences. (Hyland 1993)
Eight Japanese universities participated in the survey with 265 undergraduates responding. The questionnaire was also administered to 140 Japanese students at different English proficiency levels at a tertiary college in New Zealand. (Hyland 1994)
Essentially the concept expresses the simple idea that each learner has a clear and coherent set of learning likes and dislikes, but studies have addressed an enormously wide range of factors. (1993)
Learning style research expresses the simple idea that each learner has a clear and coherent set of learning likes and dislikes, but people differ in their learning styles in a number of ways and studies have addressed a huge range of factors. (1994)
Most of the later (1994) article is a verbatim copy of the earlier one with minor phrasing adjustments such as those shown above. This type of thing is usually known as either self-plagiarism (see the examples above) or, on a smaller scale, text recycling and is considered unacceptable in academic publishing. Many journals have rules against it, such as JALT itself:
Papers sent to JALT journal should not have been previously published, nor should they be under consideration for publication elsewhere.
I thought this was a bit strange and so I contacted JALT, the 1994 publisher, to make them aware of the issue. They told me that they take such matters very seriously and would investigate. After an investigation they informed me that this was all just an honest mistake, a bit of a mix up. The paper had been submitted to two journals by accident and and as a previous editor had dealt with the matter, it would be wrong to retract the article now.
How one accidentally submits the same article to two journals is, I must confess, a mystery to me. More mysterious was JALT’s reasoning. Regardless of what a previous editor had decided, a repeat publication is still in the literature with no indication that it is a repeat.
I wrote back suggesting that since I was unaware that it had all been resolved, as was the current editor and presumably future readers, it might be worth retracting the article, or at least adding a note to explain what had happened. They told me they were very grateful for my suggestion but no, they weren’t going to do anything. And so both articles remain in the literature.
It is also odd that Hyland himself, a incredibly respected editor and prolific author would not want the article to be retracted. In fact, until recently he continued to list both papers among his publications (his new blog, however, only lists papers from 2003). Not retracting the paper may be less embarrassing in the short term but it means that there is always the chance for some annoying blogger to bring attention at some point, to what was a mix up .
In truth, I’m not ‘that’ surprised that nothing happened here. Brendan O’Connor a student at the University of Leicester discovered that a well respected psychologist was “recycling” parts of papers into new publications to an alarming degree. Although O’Connor has documented this to an impressive degree, some journals were reluctant to do anything at all when confronted with these findings.
As whistleblowers, data sleuths and anyone else who’s contacted a scientific journal or university with allegations of impropriety will tell you, getting even a demonstrably fraudulent paper retracted is a glacial process – and that’s if you aren’t simply ignored or fobbed off by the authorities in the first place.
I was recently presented with an almost flawless piece of writing from a students whose English level precluded her producing such an almost flawless piece of writing. Initially I thought, “oh no…we have to have *that* conversation”…
In her tutorial the student guilty confessed to using translation software. I told her I was surprised because google translate famously produces awful translations from Japanese to English. “ah” she said, “I didn’t use google”.
She directed me to a site called DeepL. I threw a bit of Japanese in from Wikipedia and this is what I got out.
Now this isn’t perfect but it’s pretty damn good. For good measure I threw it into google translate and got a pretty good rendering too.
I was quite surprised at how good the Google Translate version was. But I shouldn’t have been . Sure, it was an endless source of comedy in 2004 when it produced weird and wacky sentences, but that was 15 years ago and technology moves on (in 2004 no one thought computers would beat humans at Go any time soon, that happened in 2015. There is an excellent documentary about it online). Google translate switched to using “Neural Machine Translation” around 2017 and this has reportedly led to much better quality translations.
So, is there any point in banning students from using translation software to write their essays anymore, particularly in EAP contexts? We wouldn’t mind them using dictionaries to translate words, and rather than just banning them, perhaps we could focus on getting them to use this tool more effectively? It certainly beats receiving a paid for or plagiarised submission.
So after 8 years I have decided to move to WordPress. There are a few reasons for this but I won’t bore you with the details.
You have probably noticed that the name has changed slightly. I did this because although I don’t see anything wrong with the term “Evidence based teaching”, some people do and can get a bit stuck on the idea that evidence based means “dictated by the evidence only”. I don’t see it that way but I also don’t think it is an important enough hill to die on. My twitter account will remain EBEFL, but the blog name has changed.
All of the content from the previous site should still be here but a lot of the hyperlinks no longer work. If you see one that isn’t working and draw my attention to it then I will fix it. There are over 150 posts so it could take me a while to sort this all out!