Completed projects

Aptis logo

Exploring rater interaction with test-taker responses in Aptis Writing

Researcher: Nathaniel Owen

Project summary

This project investigates expert judgement in assessing student writing in English language tests. The Aptis test is produced by the British Council and is used by a variety of companies and education bodies. Test takers are judged on their English language proficiency. Raters compare test taker writing to a score rubric and assign a mark accordingly. The study explores how Aptis raters engage with writing scripts produced by test-takers and which elements of the marking rubrics influence their decision-making. The study uses innovative eye-tracking technology to identify which parts of the rubric and how much text they read before reaching a decision.

Information about the British Council Aptis test can be found here: www.britishcouncil.org/exam/aptis/writing

Outcomes

Findings from the study will be fed back to the British Council, who will use them to inform future rater training and descriptors for evaluating test-taker writing. The findings of the study will also be used to provide recommendations regarding the content of the scoring rubrics; to what extent are raters marking analytically or holistically? Are parts of the rubrics for some tasks being overlooked in favour of others? Outcomes will also include additional validation evidence that specific tasks target specific cognitive writing skills.

TOEFL.iBT logo

Researching Academic Reading in two contrasting EMI (English as a Medium of Instruction) university contexts, and implications for the design of TOEFL iBT

Researchers: Nathaniel Owen, Prithvi Shreshtha, Kristina Hultgren, Stephen Bax

Project summary

This project investigates the use of English in the education systems of two countries – Sweden and Nepal. These countries attracted interest as they use English widely in their education systems despite English not being their native language. The aim of the research is to find out how English is used in these contexts at university level, focusing specifically on reading materials, and whether the TOEFL test is suitable to make decisions about whether students are ready to study in English.

The TOEFL test (Test of English as a foreign language) is used widely by universities in English-speaking countries as evidence that incoming international students are able to cope with the language demands of their academic programmes. More recently, this test is being used in other contexts, such as in Nepal and Sweden, to ensure that students in those countries can cope with their English-language curricula. This study will contribute to understanding whether the test is suitable for this purpose, as this new use represents an extension to its mandate.

Further information about the TOEFL test can be found here: www.ets.org/toefl

Outcomes

The project will offer important new insights into academic reading in EMI contexts, and more specifically will provide important new insights into the potential role of TOEFL iBT in EMI contexts to underpin future development and expansion of the TOEFL.

Aptis logo

Researching lexical thresholds and lexical profiles across the Common European Framework of Reference for Language (CEFR) levels assessed in the Aptis test

Researchers: Stephen Bax, Prithvi Shreshtha, Nathaniel Owen

Project summary

Major language exam boards carry out extensive research into the content of their tests to ensure that this is fit-for-purpose. This might be the texts used in reading tests for example, which are investigated to determine the suitability of vocabulary. Additionally, these exam boards are also concerned with the use of language produced by candidates in writing and speaking portions of their tests. This research project is concerned with the writing portion of the Aptis test. The Aptis test is produced by the British Council and is used by a variety of companies and education bodies to make claims about students or employees. The Aptis test is designed to be able to differentiate between a wide range of language abilities, from beginner to advanced.

One way of investigating whether a test is able to differentiate between a range of abilities is by empirical investigation comparing test-taker language against an existing language framework such as the CEFR (Common European Framework of Reference). The CEFR is designed to be a “transparent, coherent and comprehensive basis for the elaboration of language syllabuses and curriculum guidelines, the design of teaching and learning materials, and the assessment of foreign language proficiency” (www.coe.int/en/web/common-european-framework-reference-languages/home). The CEFR contains a number of levels of language proficiency, with descriptors of what language students at each level are capable of. The score bands of the Aptis test were designed to align to the levels of the CEFR. Language produced in test conditions by large numbers of test-takers can be compared against these descriptors to see whether the test has elicited language from which similar language proficiency claims can be made.

This study investigates whether these levels can be described through descriptive elements of student writing, such as vocabulary diversity, range and sophistication. The study uses automated analysis tools found in Text Inspector (www.textinspector.com). 6,000 transcripts of student writing were processed to obtain a range of language variables, benchmarked against the scores they received in the Aptis test. The metrics are being analyse statistically to see whether these are significant differences across the score bands.

Outcomes

The result will be a comprehensive and detailed picture of the language used by Aptis test takers across the Common European Framework of Reference for Language (CEFR) levels assessed in Aptis. This will provide evidence that the Aptis test is sufficiently finely-grained to distinguish between different language abilities.

WiSP logo

Writing in professional social work practice in a changing communicative landscape (WiSP)

Researchers: Theresa Lillis, Maria Leedham, Alison Twiner, Lucy Gray

Project summary

Writing is a key part of every social worker’s job – case notes, assessment reports, emails, letters, support plans… There is a lot debate about the amount of writing being done (is too much time being spent on ‘recording’?) and the quality of written records. However, little empirical research has been carried out on writing in social work practice. The WiSP project seeks to fill that gap by language researchers working closely with social workers and social work agencies.

For more details, see the project website: www.writinginsocialwork.com

Fake news cartoon

Creating Facebook

Researchers: Philip Seargeant, Caroline Tagg

Project summary

Creating Facebook is a project run by Philip Seargeant and Caroline Tagg of the Open University’s Applied Linguistics and Literacies (ALL) Research Group. Amy Aisha Brown is research associate.

The project investigates people’s awareness of how communication via Facebook works, and the social implications of their communication. This includes the way that information and opinions are circulated, and how people react to them.

 

Conceptualisations of Academic Writing in the University of Technology Sector

Project summary

Conceptualisations of Academic Writing in the University of Technology Sector is a joint project funded by the Cape Peninsula University of Technology (CPUT), a vocationally-focused higher education institution in South Africa. The two-year project involves collaboration between Dr Lynn Coleman of CPUT and Dr Jackie Tuck of The Open University.

The project explores lecturers’ perspectives on student writers and their writing, and how their understanding of students' work shapes teaching practice on the ground. We focus on lecturers in Management Studies disciplines at a vocational higher education institution in South Africa, including multiple interviews with individual lecturers around student texts. We also gathered institutional and pedagogic texts and examples of student work shared by participants. Results show that while teachers still persist in framing students negatively as being 'deficit' in academic literacy skills, these dominant framings are also challenged in the practice of some lecturers. The study will contribute to debates about how institutions can best support transformative pedagogies of academic communication and will inform the field of academic development.

 

Living with Uncertainty: Metaphor and empathy

Project summary

This work includes research which is part of the RCUK Global Uncertainties programme and looks at empathy between individuals and social groups in times of uncertainty and increased violence.

Metaphor is viewed in terms of multiple interconnected dimensions- linguistic, cognitive, affective, physical and cultural. Metaphor and discourse dynamics analysis have been used to investigate data from the UK, Brazil and the USA.

The research has resulted in a multi-level dynamic model of empathy.  This understanding of empathy has informed Lynne Cameron’s visit to Maralal in northern Kenya to explore conflict transformation activities that are underway in that region.

For further details see the project web site Empathy in making change happen.

To find out more about the experience of undertaking the project and the latest developments go to Lynne Cameron’s blog.