Analysing my Feedback Language

TL:DR SUMMARY

I ran a feedback text I’d written on a student’s work through some online text analysis tools to check the CEFR levels of my language. I was surprised that I was using some vocabulary above my students’ level. After considering whether I can nonetheless expect them to understand my comments, I propose the following tips:

• Check the language of feedback comments before returning work and modify vocabulary necessary.
• Check the vocabulary frequently used in feedback comments, and plan to teach these explicitly.
• Get students to reflect on and respond to feedback to check understanding.

A couple of colleagues I follow on blogs and social media have recently posted about online text analysis tools such as Text Inspector, Lex Tutor and so on (see, for example Julie Moore’s post here and Pete Clements’ post here). That prompted me to explore uses of those tools in more detail for my own work – both using them to judge the input in my teaching materials or assessments, and also using them with students to review their academic essay writing.

Once I got into playing around with different online tools (beyond my go-to Vocab Kitchen), I wanted to try some out on my own texts. The thing I’ve been writing most recently, though, is feedback on my students’ essays and summaries. But, I’m a bit of a feedback nerd so I was quite excited when the idea struck me: I could use these tools to analyse my language in the feedback I write to help my students improve their texts. A little action research, if you will. 

Now I obviously can’t share the students work here for privacy and copyright reasons, but one recent assessment task was to write a 200-250 word compare/contrast paragraph to answer this question:

How similar are the two main characters in the last film you watched?

(Don’t focus on their appearance).

These students are at B2+ level (CEFR) working towards C1 in my essay writing class. They need to demonstrate C1-level language in order to pass the class assessments. One student did not pass this assessment because her text included too many language mistakes that impeded comprehension, because overall the language level did not reach C1, and because she didn’t employ the structural elements we had trained in class.

Here’s the feedback I gave on the piece of work and which I ran through a couple of text checkers. (Note: I usually only write this much if there are a lot of points that need improving!)

The language of this text demonstrates a B2 level of competence. Some of the phrasing is rather too colloquial for written academic language, e.g. starting sentences with ‘but’, and including contracted forms. You need to aim for more sophisticated vocabulary and more lexical diversity. More connectors, signposting and transitions are needed to highlight the genre and the comp/cont relationships between the pieces of information. The language slips lead to meaning not always being emphasised or even made clear (especially towards the end). Aim to write more concisely and precisely, otherwise your text sounds too much like a superficial, subjective summary.

Apart from the personal phrase at the beginning, the TS does an OK job at answering the question of ‘how similar’, and naming the features to be discussed. However, you need to make sure you name the items – i.e. the characters – and the film. In fact, the characters are not named anywhere in the text! The paragraph body does include some points that seem relevant, but the ordering would be more logical if you used signposting and the MEEE technique. For example, you first mention their goals but don’t yet explain what they are, instead first mentioning a difference between them– but not in enough detail to make sense to a reader who maybe doesn’t know the series. Also, you need to discuss the features/points in the order you introduce them in the TS – ‘ambition’ is not discussed here. The information in the last couple o sentences is not really relevant to this question, and does not function as a conclusion to summarise your overall message (i.e. that they are more similar than they think). In future, aim for more detailed explanations of content and use the MEEE technique within one of the structures we covered in class. And remember: do not start new lines within one paragraph – it should be one chunk of text.

I was quite surprised by this ‘scorecard’ summarising the analysis of the lexis in my feedback on Text Inspector – C2 CEFR level, 14% of words on the AWL, and an overall score of 72% “with 100% indicating a high level native speaker academic text.” (Text Inspector). Oops! I didn’t think I was using that high a level of academic lexis. The student can clearly be forgiven if she’s not able to improve further based on this feedback that might be over her head! 

(From Text Inspector)

In their analyses, both Text Inspector and Vocab Kitchen categorise words in the text by CEFR level. In my case, there were some ‘off list’ words, too. These include abbreviations, most of which I expect my students to know, such as e.g., and acronyms we’ve been using in class, such as MEEE (=Message, Explanation, Examples, Evaluation). Some other words are ‘off list’ because of my British English spelling with -ise (emphasise, summarise – B2 and C1 respectively). And some words aren’t included on the word lists used by these tools, presumably due to being highly infrequent and thus categorised as ‘beyond’ C2 level. I did check the CEFR levels that the other ‘off list’ words are listed as in learners’ dictionaries but only found rankings for these words: 

Chunk – C1

Genre – B2

Signposting – C1

(From Vocab Kitchen)

Logically, the question I asked myself at this point is whether I can reasonably expect my students to understand the vocabulary which is above their current language level when I use it in feedback comments. This particularly applies to the words that are typically categorised as C2, which on both platforms were contracted, superficial and transitions, and perhaps also to competence, diversity and subjective which are marked as C1 level. And, of course, to the other ‘off list’ words: colloquial, concisely, connectors, lexical, and phrasing.

Now competence, diversity, lexical and subjective shouldn’t pose too much of a problem for my students, as those words are very similar in German (Kompetenz, Diversität, lexikalisch, subjektiv) which all of my students speak, most of them as an L1. We have also already discussed contracted forms, signposting and transitions on the course, so I have to assume my students understand those. Thus, I’m left with colloquial, concisely, connectors, phrasing and superficial as potentially non-understandable words in my feedback. 

Of course, this feedback is given in written form, so you could argue that students will be able to look up any unknown vocabulary in order to understand my comments and know what to maybe do differently in future.  But I worry that not all students would actually bother to do so –  so they would continue to not fully understand my feedback, making it rather a waste of my time having written it for them.

Overall, I’d say that formulations of helpful feedback comments for my EAP students need to strike a balance. They should mainly use level-appropriate language in terms of vocabulary and phrasing so that the students can comprehend what they need to keep doing or work on improving. Also, they should probably use some academic terms to model them for the students and make matching the feedback to the grading matrices more explicit. Perhaps the potentially non-understandable words in my feedback can be classified as working towards the second of these aims. 

Indeed, writing in a formal register to avoid colloquialisms, and aiming for depth and detail to avoid superficiality are key considerations in academic writing. As are writing in concise phrases and connecting them logically. Thus, I’m fairly sure I have used these potentially non-understandable words in my teaching on this course.But so far we haven’t done any vocabulary training specifically focused on these terms. If I need to use them in my feedback though, then, the students do need to understand them in some way. 

So, what can I do? I think there are a couple of options for me going forward which can help me to provide constructive feedback in a manner which models academic language but is nonetheless accessible to the students at the level they are working at. These are ideas that I can apply to my own practice,  but that other teachers might also like to try out:

• Check the language of feedback comments before returning work (with feedback) to students; modify vocabulary if necessary.
• Check the vocabulary items and metalanguage I want/need to use in feedback comments, and in grading matrices (if provided to students), and plan to teach these words if they’re beyond students’ general level.
• Use the same kinds of vocabulary in feedback comments as in oral explanations of models and in teaching, to increase students’ familiarity with it. 
• Give examples (or highlight them in the student’s work) of what exactly I mean with certain words.
• Get students to reflect on the feedback they receive and make an ‘action plan’ or list of points to keep in mind in future – which will show they have understood and been able to digest the feedback.

If you have further suggestions, please do share them in the comments section below!

As a brief closing comment, I just want to  point out here that it is of course not only the vocabulary of any text or feedback comment that determines how understandable it is at which levels. It’s a start, perhaps, but other readability scores need to be taken into account, too. I’ll aim to explore these in a separate blog post.

Advertisement

Vocabulary lists: snog, marry, avoid? – A summary

I recently wrote a post for ELT Research Bites summarising a research article on ‘Is There a Core General Vocabulary? Introducing the New General Service List’, which introduced a new vocabulary list that, the authors propose, can inform vocabulary instruction in ELT. I thought undersdanding how such lists are put together would help teachers and writers make better decisions on how to use such vocabulary lists in their classrooms or materials. And so I was attracted to Julie Moore’s talk at IATEFL in Brighton, entitled “Vocabulary lists: snog, marry, avoid?” to see if my thinking was in line with others who are more knowledgeable than me in such things! Let me tell you more about what Julie said… 

To start, we got a useful reminder of what vocabulary lists are: published, standrdised lists of words, phrases or chunks, based on a certain frequency criteria and usually intened for use with students. Examples Julie gave included the GSL, Oxford 3000, and the Phrasal Verbs List. Some lists are more specialised, e.g. the AWL, and others take a slightly different approach in that they try to group vocabualary items into levels at which language learners can be expected to learn them (e.g. English Vocabulary Profile, GSE). 

Why are vocabulary lists useful? 

Julie explained that the items on the lists are selected based on certain principles (which depend on the list) and can thus form a principled basis for devising vocabulary teaching syllabi. For those of us working with the language, she said, they can be a useful tool for confirming our inuitions about the frequency of words and, in ELT, their level appropriacy. They mean we don’t have to start from scratch every time we wish to compose a vocabulary teaching syllabus or material, for example using corpora or other souces to collate data about words’ frequency before we select items for inclusion. Also in this context, if different people are working on different parts of a material or syllabus, for example, using a vocbulary list to guide the language used and presented can help maintain consistency across the parts. 

But… 

Not understanding the concept of vocabulary lists or the selection criteria and data used to compile them can make using them frustrating or even downright misleading! There are some key issues that can make compiling such vocabulary lists, and then working with them, problematic. Julie mentioned a few, such as decisions on what items to count for frequency (words, lemmas, chunks, word families?) and which sense of a polyseme to use for selection or level-categorisation. In Julie’s words, the nature of the English language is “messy” and contains numerous obstacles for anyone attempting to represent it in as straightforward a form as a list! As teachers, we also know that the progression of language learning is messy, too: it’s a non-linear porocess, including learning words for active and/or receptive use, which will therefore be difficult to tame into a list! Especially regarding lists like the GSE or EVP as ‘levelling tools’, Julie reminded us (though who could forget?) about differences between learners, for example their main language, which will also affect how easy they find certain vocabulary items – and which these lists cannot possibly take into account! 

So…

Julie’s call to arms echoes my own sentiments: Do not blindly believe everything a vocabulary list seems to show you! They can be very helpful, if the right list is used wisely and for the right purpose! Vocabulary lists are one tool of many at our disposal when we need to make decisions on what to include in our syllabi, teaching or materials, alongside things such as dictionaries and our knowledge of the target learners. And the different lists were composed through different selection methodologies, from different data, and with different aims, making them more appropriate for certain purposes than others. So whether you ultimately decide to snog, marry or avoid the next handsome vocabulary list you meet, take your time to get to know it first!

Worksheet-free Vocab Revision Activities

What do you do in those last 5 minutes of class when you’ve finished everything that was planned? Or when energy levels hit a low during a lesson? Or in that lull while the next student gets ready to present, or whatever? We all know about the need to revise and recycle new vocabulary in language lessons, and in this post I want to share a few vocabulary revision activities that teachers can slot into any downtime that might occur in a lesson!

I’ve built up my repertoire of this kind of quick review activity over the years, so many are borrowed or adapted from colleagues, and others are based on popular board games. I want to give you a collection, all in one place, of collaborative and competitive activities that check students have remembered and actually understood new words (i.e. there are no rote learning activities here!) You can print out this post and take it to lessons with you – that’s the only paper you’ll need: all of these activities have one main thing in common – you don’t need to photocopy anything to do them!

1. Scategories

Choose a category of vocabulary you want students to revise, for example ‘character traits’, ‘school subjects’, ‘transition words showing contrast’. Choose 5-10 letters of the alphabet and write them, with the category, on the board. Students (in teams, if you wish) now have 1 minute to come up with one vocabulary item fitting the category which starts with each of the letters you have chosen. Compare answers. To make it into a competition, give points: Students or teams get 2 points if they’ve written a correct vocab item that no one else / no other team has written, and one point for correct vocab items that someone else wrote down, too.

2. ‘Taboo’ on the board

Like the game ‘Taboo’, but without any little slips of paper that need preparing! It works best with nouns. Get your learners to sit with their backs to the board. Option 1: Choose one student to look at the board and see the word you’ve written there. They have to explain it to the other students, who try to guess which word is being explained. The first student who guesses correctly can be the next one to explain a word. Option 2: Group competition! Students sit in teams/groups with their backs to the board. One team member turns around and looks at the word you’ve written on the board, and explains it to their team members, who try to guess which word it is. Give them a time limit (e.g. 30 secs per word). For each word correctly guessed within the time limit, the team gets one point (keep track on the board) and then the next team has a turn. To make either option more difficult, write the main word on the board (maybe put a circle around it) and add two or three ‘taboo’ words which are not allowed to be used in the explanation. For example, if the main word is “bauble”, the taboo words might be “Christmas,” “tree” and “decoration.”

3. Beep

This guessing game works best with verbs or verb phrases, but nouns can be good, too. One student is told a ‘secret word’ which is to be ‘beeped out’ (like swearwords on TV). The other students ask them yes/no questions to try to guess the secret word – each student is only allowed one question at a time. For example, “Who BEEPS?” “Do you BEEP on your own?” “What do people BEEP most often?”  As these examples show, the activity can be used with fairly low-level language, but I’ve also used it in EAP with verbs such as research, evaluate, and analyse. After their question has been answered, the student can make a guess at the secret word, if they wish – if they get it right, they can be the next one who is given a secret word. To make it more difficult, allow each student only 2 guesses at the secret word during each round.

4. Sentence editing bingo

I like using this one to revise adverbs or adverbial phrases, but nouns work, too. Students are asked to write down a number of vocab items that you’ve recently covered in a particular category (e.g. adverbs of manner, adverbial phrases for time/place, things you find in a classroom). Choose the number according to how much time you have and how many sentences you think you’ll get through. Usually 5 or so is enough. Students can also work in pairs. Write a simple sentence on the board, such as “I like reading.” Students tick off one of their words if they think it can fit correctly into the sentence. For example, a student might tick off ‘in the evening’ or ‘really,’ or maybe ‘books’ if you’ve gone with nouns. Repeat this with several sentences. Once a student has ticked off, i.e. thinks they’ve been able to use appropriately, all of their words/phrases, they shout ‘Bingo!’ Check their answers together as a class – if there’s time, check other students’ suggestions, too.

5. Changing corners

This activity will get students up and moving around the room! Make sure they move their chairs and bags out of the way! Nominate corners or sides of the room that are the ‘spelling zone’, ‘definition zone’,  and ‘example zone’. Call out one vocabulary item you want to revise. Students have to move and stand by the corner or wall that shows the challenge they feel comfortable doing with that word: spelling it, defining it, or using it in an example sentence. Pick one student from each zone to give their answer out loud. To make it a competition, either give points for correct answers (1 for spelling, 2 for defining, 3 for an example use), or get anyone who gives an incorrect answer to sit down, then keep going with different vocab items until only three students are left! (For this, you might need to increase the difficulty of the words as you go along!)