Teacher Education for Language Assessment and Testing: Postgraduate Program Evaluation from its Students’ Perspective

Language assessment and testing (LAT)

The history of LAT has been recorded by researchers such as Spolsky (1995, 2017) and Farhady (2018) . As for the history of testing, Spolsky (2017) said that, looking back over the last halfcentury, LAT has developed into an identifiable academic field, as well as into a major industry. Through most of its history, it was practiced mainly in the form of summative assessment, that is, inclass tests and final class-based tests or external examinations ( Spolsky, 2017. Farhady, 2018 ). With technological development, testing also moved on from pen and paper to different forms of ComputerBased Tests (CBT), including Computer Adaptive Tests (CAT) ( Chapelle and Douglas, 2006 ).

In recent years, there has been an interest and focus on FA, conducted in different forms such as projects, presentations, group activities, ePortfolios, and games ( Stiggins, 2002 ). With all these developments, it was of utmost importance to define language teachers’ LAL and LAL competences, as well as the way in which these are reflected on LAT TE courses. However, although research has been conducted in different aspects related to teacher LAL and training, not much has been done so far in finding how this training is perceived by one of its major stakeholders, that of the students who take the TE course.

Language assessment literacy (LAL): definition and competences

There has been much debate on defining LAL and developing a common framework of what LAL entails. The aim was for this definition to constitute part of training programs and the repertoire of language teachers’ assessment and testing practices.

According to Webb (2002, p. 3) , “Assessment Literacy is defined as the knowledge of how to assess what students know and can do, interpret the results of these assessments, and apply these results to improve student learning and program effectiveness”. LAL broadly refers to what language teachers and other stakeholders need to know about language assessment matters and activities ( Stiggins and Conklin, 1992. Inbar-Lourie, 2008. Malone, 2008. Taylor, 2013. Fulcher, 2012. Jeong, 2013. Lam, 2015. Tsagari and Vogt, 2107. Nimehchisalem and Bhatti, 2019). Herrera and Macías (2015, p. 305) added that “High LAL competence should enable EFL teachers to design appropriate assessments, select from a wider repertoire of assessment alternatives, critically examine the impact of standardized tests (e.g., TOEFL, IELTS, etc.), and establish a solid connection between their language teaching approaches and assessment practices”.

Brindley (2001) was one of the first to come up with a LAL framework. It was intended for language teachers and acknowledged their different needs. Together with the proficiency standards, the Common European Framework of Reference (CEFR) (Council of Europe, 2001) also introduced and promoted alternative forms of assessment ( Inbar-Lourie, 2017 ), thus influencing LAT TE.

Boyles (2006) defined LAL as an understanding of the principles and practices of testing and assessment in order to be able to analyze them. She recommended the development of “a universal understanding of what constitutes a good assessment and to build a common, articulated set of criteria for exemplary assessments” (p. 22). She supported teacher development and recommended it should be ongoing, both face-to-face and online, and in different contexts such as school districts, professional meetings and Summer Institutes, online professional development, workshops, preservice programs at undergraduate level, and online resources.

Davies (2008) proposed a very useful LAL dimensions core list: knowledge, skills, and principles for language teachers. Giraldo (2018, see p. 186188) expanded on that, and, based on the existing literature ( Coombe et al., 2012. Davies, 2008. Fulcher, 2012. Inbar-Lourie, 2008. Malone, 2013. Scarino, 2013. Taylor, 2013 ), he proposed a set of descriptors for knowledge, skills, and principles in eight dimensions of LAL for language teachers. Researchers such as Jeong (2013) supported that LAL may be different to different stakeholders, for example, language testers and non-language testers. The review of LAL definitions, frameworks, sets of competences, and LAL dimensions and descriptors recorded the development in the area and the debate, which set the background for the LAT course content.

Language assessment course content

As a continuum to the above, other studies focus on examining the content of TE courses to find out whether and how they reflect LAL discussed and suggested by different researchers. Bailey and Brown conducted the first study on LAT courses in 1996. The purpose was to investigate the instructors’ background, the topics covered, and students’ apparent attitudes towards those courses. They repeated this research in 2007 to examine the same characteristics and how such courses may have changed since 1996. The results described the instructors, the course characteristics, and the students, as well as the differences and similarities between the 1996 and 2007 results ( Brown and Bailey, 2008 ). None of these studies reported on course evaluation from the students’ perspective.

In a nationwide survey of 86 instructors, Jin (2010) investigated the training of tertiary level FL teachers in China, focusing on language testing and assessment courses. More specifically, “the results revealed that the courses adequately covered essential aspects of theory and practice of language testing” ( Jin, 2010, p. 555). The teaching content of the different types of courses were similar to a great extent. Suggestions for improvement included highlighting some under-addressed content aspects and setting up a network of teacher-testers for the exchange of experiences, professional knowledge, and skills. Similar to Brown and Bailey’s studies (1996 and 2008), Jin’s study (2010) did not focus on the trainees’ point of view regarding the course.

Acknowledging the different understanding of LAL by FL instructors and language testing experts in the USA, Malone (2013) analyzed how course participants and professional external observers felt about course contents. She concluded that, when it came to preferred content in LAT TE courses, language test experts opted more on theoretical aspects, whereas language teachers opted more for testing tasks.

Giraldo and Murcia (2018) reported on the preliminary findings of their study, which aimed to identify the impact of a LAT course for preservice teachers in a language teaching program in a Colombian state university. Their findings indicated “that there is a need to combine theory and practice of language assessment, with an emphasis on current methodologies for language teaching, assessment in bilingual education, and local policies for assessment” (p. 1). Regarding LAT course content, the gap identified in the research conducted so far was the lack of studies on the view and evaluation of LAT TE courses by LAT TE participating students.

Need for LAL teacher education

Language teachers spend up to a third of their time in assessment-related activities ( Cheng et al., 2001 ). In their language teaching daily practices, they have to deal not only with their own classroombased assessment, but also with standardized language tests ( Mede and Atay, 2017 ). Despite this, most of them “do so with little or no professional training” ( Bachman, 2000, p. 19-20). In addition, most of their knowledge and practice in the area focus on testing and does not include alternative assessment types. Here are some examples of this focus on testing: according to Inbar-Lourie (2017) , there have been two sources which disseminate language testing knowledge, namely language testing textbooks and language testing courses.

In 2008, Davies reviewed language testing textbooks and found that their approach to language testing mainly covered two out of the three core list LAL dimensions he suggested: knowledge and skills, as well as principles to a lesser extent. This was reaffirmed in the case of language testing courses ( Bailey and Brown, 1996. Brown and Bailey, 2008). LAT university courses constitute compulsory or elective courses in graduate and doctoral programs. They are mainly in the area of applied linguistics, English for speakers of other languages, English as a second or FL, educational linguistics, or CALL ( Language Assessment, 2020 ).

Universities worldwide offer regular courses and programs that focus on LAT TE at both PhD and MA levels. In the USA, for example, such programs are offered by the Iowa State University, the University of Illinois, Urbana-Champaign, the University of Hawai’i at Manoa, Teachers College, Columbia University, Penn State University, Georgia State University, Northern Arizona University, McGill University, and the University of Toronto. Lancaster University, as well as the Universities of Leicester, Bristol, Cambridge, and Bedfordshire offer such courses in the UK, and, in China, the Guangdong University of Foreign Studies. At the MA level, indicative universities include Lancaster University and University of Leicester in the UK, as well as California State Universities at Fullerton, Los Angeles, Long Beach, San Jose, and San Francisco in the USA ( Language Assessment, 2020 ).

The focus of most such courses has been on summative assessment, most specifically on test development and the different aspects evolving around it:

test development, psychometric qualities of tests, validity, reliability and fairness of tests, washback, and classical true score measurement theory. Others also focus on item response theory, factor analysis, structural equation modelling, G theory, latent growth modelling, qualitative analysis of test performance data such as conversation and discourse analysis, and politics and language policy issues (Language Assessment, 2020).

While more attention has been paid to Davies’ category of principles, there has also been a shift away from a testing-oriented LAL focus ( InbarLourie, 2017 ) to contextually relevant and diverse practices ( Inbar-Lourie, 2008 ) and FA ( Stiggins, 2002 ). At the theoretical level, sociocultural approaches such as Vygotski’s take on language learning and evaluation also had their influence on LAT ( Infante and Poehner, 2019 ). All these were expected to be reflected on LAT TE courses. Moreover, researchers such as Malone (2011) and Purpura and Turner (2013) supported that teaching and assessment should become integrated in order to inform and improve one another.

Although there are language teacher textbooks and training courses, it has been noted that there is also an imbalance between LAT practices at schools and LAT courses at universities, and that teachers’ and lecturers’ training is inadequate ( Jeong, 2013. Lam, 2015. Kalajahi and Abdullah, 2016. Hadigol and Kolobandy, 2019 ). It has also been noted that teachers’ knowledge depends on their assessment identities and is based on their prior experiences, beliefs, and feelings ( Looney et al., 2017 ) rather than their training. The need for LAT training was also raised by other researchers such as Volante and Fazio (2007) and Scarino (2013) .

Pehlivan Şişman and Büyükkarci (2019) noticed that “in the last decades, LAL has been viewed as one of the fundamental competencies of a language teacher” (p. 629). Therefore, more attention has been given to research LAL TE. Research in this regard has attempted to establish LAL TE practices and their impact on language teachers. Findings related to LAT postgraduate studies have been discussed, and other studies have focused on undergraduate programs. In Colombia, for example, López and Bernal (2009) examined teachers’ perceptions about LA and the way they use language assessments in their classroom. They found that there is a significant difference in their perceptions, depending on their level of training in LA. Amongst other findings, they also found that, out of 27 undergraduate programs, only seven of them included a course on evaluation. The authors highlighted the importance of providing adequate LAT TE for all prospective language teachers in Colombia.

Lam (2015) examined LAT TE in five Hong Kong institutions. The findings indicated that LAT training in Hong Kong remains inadequate, as “pre-service teachers are usually ill-prepared to assess their own students owing to inadequate training provided by teacher education programmes” (p. 3). Jeong (2013) examined LAT courses and found significant differences in the topics covered, as well as a difference in the interests of language testers and non-language testers teaching LAT courses. For this reason, Jeong called for a common understanding of LAL among the courses.

Vogt and Tsagari (2014) examined the level of FL teachers in LAL in selected countries across Europe. Their results indicate “that only certain aspects of teachers’ Language testing and assessment expertise are developed” (p. 374). For this reason, teachers tend to learn “on the job or use teaching materials for their assessment purposes” (p. 374). Although priorities varied depending on each country’s educational context, teachers identified the need for training in language testing and assessment aspects.

Tsagari and Vogt (2017) also examined how FL school teachers’ perceived LAL levels in Cyprus, Germany, and Greece. They reported that their sample teachers were not adequately prepared to conduct assessment-related tasks, as these teachers had no training. Sultana (2019) charted the area of LAL for English language teachers in Bangladesh. The study results revealed the teachers’ inadequate academic and professional testing background, a fact that hindered their LAT practices. Sultana made suggestions for the development of LAT awareness of English teachers in Bangladesh.

Nimehchisalem and Bhatti (2019) conducted a literature review on LAL in the last two decades (19992018). This review revealed eight research studies on the topic ( Volante, 2007. Lam, 2015. Deneen and Brown, 2016 , Razavipour et al., 2011; O’Loughlin, 2013. Malone, 2013. Jeong, 2013 ). Nimehchisalem and Bhatti argued that, although there has been more interest in classroom assessment, according to research, many teachers and instructors were not adequately trained when they started teaching. In other words, they did not have the necessary knowledge regarding assessment, which could help them in the development, administration, and interpretation of tests.

Malone (2008) made an effort to define adequate LAT training. To do so, he argued that, in order for it to be adequate, training should include “the necessary content for language instructors to apply what they have learned in the classroom and understand the available resources to supplement their formal training when they enter the classroom” (p. 235). Herrera and Macías (2015) moved on to say that

good practices in EFL assessment should be modelled by teacher educators throughout the programme curriculum, making explicit assessment expertise in the courses. In this way, prospective EFL teachers will recognise the assessment practices teacher educators use and will start building their personal knowledge based on LA as informed by their experiences in EFL TE programmes and by the content of assessment courses. Such a personal knowledge base has the potential to be progressively refined as they advance in their careers as in-service language teachers. (p. 310)

In conclusion, although there has been an increased interest in FA, the focus of most LAT TE programs has so far mainly been on summative assessment. As a result, students and future language practitioners are not adequately prepared for what they are required to deal with in their daily lives as professionals. Although there are LAT courses, as Inbar-Lourie (2017) admits, language teachers lacking assessment practices are “often the result of inadequate or non-existent training” (p. 267). Most importantly, although they constitute “the largest group of LAL stakeholders”, language teachers “are seldom listened to in the LAL debate” (p. 267). Scarino (2013) also emphasized the importance of language teachers as the most important of all the stakeholders, given that they are the direct test users.

Language assessment TE courses from the teacher educator perspective

Research has been conducted to examine teacher educators’ perspective, that is, what these instructors think about LAT TE courses and how they evaluate them. Jeong (2013) investigated the effect brought about by instructors in shaping the characteristics (i.e., content and structure) of LAT TE courses. Her research results indicated the existence of significant differences in the content of the courses, depending on the instructors’ background in six topic areas: test specifications, test theory, basic statistics, classroom assessment, rubric development, and test accommodation. Course similarities and differences were identified. The importance of a common LAL understanding among all stakeholders was acknowledged as an area to be addressed.

Language teachers’ and LAT TE participating students’ perspective

Research reports that few research studies involve language teachers in the area of language assessment, whose complex variables of LA include educational and assessment policies in and outside institutions, the dynamics of the classroom, and what practitioners bring into the assessment process. Inbar-Lourie (2017) , for example, argues that a limited number of research studies involve teachers’ voices and perspectives. The. Scarino (2013) describes cases which help in the creation of a personalized LAL knowledge base, stemming from language teachers’ own experiences, beliefs, suppositions, and understanding of assessment. Csépes (2014) reports the reluctance of language teachers to adopt alternative assessment procedures suggested by government policies. Hatipoğlu (2015) examines 124 pre-service English language teachers’ perceptions of their prior knowledge on language testing and what a LAT course should include prior to course development. The findings revealed “the effect of local assessment cultures and previous assessment experiences on pre-service teachers’ perceived needs related to language assessment literacy” (p. 111). This study highlighted the importance of lecturers’ collaboration with students during course development.

The above literature review revealed that there is only one study focused on the perceptions of LAT TE participating students. It focuses on their perceptions of language testing prior knowledge and what a LAT course should entail. To the authors’ knowledge, no research has dealt with the perceptions of LAT TE courses regarding their content and value after completion, nor has there been a focus on the evaluation of such courses from the participants’ point of view. These are serious matters that need to be investigated in depth in order to enrich the literature and fill the gap.

The CALAT LA TE course

The Computer-Assisted Language Assessment and Testing (CALAT) module was one of the eight online modules taught in the CALL program, which are offered by a university in Cyprus. It was a response to the need for LAT TE programs, which would reflect current theories and practices in the area of LAT and would cater for the needs of current LAT classroom practices. The 13-week module was delivered online via the Moodle Platform. Other technologies included Google Drive, email, Facebook, Messenger, Internet resources, Kahoot, etc. The module covered many areas covered in Davies’ (2008) LAL dimensions core list and in Giraldo’s (2018) proposed set of descriptors. It was based on constructivism and social constructivism theories of learning. Assessment was integrated in the module in order to inform and improve participants’ assessment practices ( Malone, 2011 ; Purpura & Turner, 2013). This included quizzes, self, peer- and instructor feedback, ePortfolios, rubrics, and a digital final examination. The module content included “the necessary content for language instructors to apply what they have learned in the classroom and understand the available resources to supplement their formal training when they enter the classroom” ( Malone, 2008, p. 235). LAT practices were suggested, such as the ones mentioned earlier, and the use of technologies for course delivery and assessment. The learning theories were modeled by the instructor throughout the course. The aim was to help language teachers recognize the assessment practice types used during the module in the form of content, LAT knowledge, skills and experiences acquired, and LAT practices they experienced through LAT artefact construction, as well as the construction and application of assessment types in their language teaching practice. The long-term aim was for such personal knowledge to have the potential to be progressively refined as the participants advanced in their careers ( Herrera and Macías, 2015 ).

Time of implementation of program evaluation from the students’ perspective

The MA in CALL program was first offered in September 2015. At the end of the academic year 2018-2019, which marked the end of the first 4-year period (June 2019), a comprehensive program evaluation from the students’ perspective was carried out. This involved all the students who attended the program during those four years.

CALAT TE participating students

There was no process for selecting the participants due to the small number of students who attended the course. For this reason, the respondents were chosen based on their availability ( Babbie, 1990 ). All 25 students who participated in the CUT MA in CALL program over a four-year period (2015-2019) were invited to participate in the research. Participation was voluntary. Nineteen (76%) of the total population participated in the research. Their ages ranged between 22 and 60. The GPA mean was 2,80 for females, and 2,52 for males out of a possible 4,00. The participants’ L1 included English, Greek, Dutch, and Igbo. Participants had learned English as a FL, and all of them had an advanced proficiency level in English (CEFR B2 to C2). The CALAT module participants were practicing language teachers of different languages, namely English (8), Greek (6), Turkish (2), German (1), Igbo (1), and French (1), at different education levels such as primary (3), secondary (14) and tertiary (2), in different countries: Cyprus (10), Greece (6), the Netherlands (2), and Nigeria (1).

Data collection instruments

A Google Form questionnaire (see Appendix 1) was distributed to all the 25 CALAT TE course participants who were from four different yearly intakes ( Nesbary, 2000. Sue and Ritter, 2007 ). The response rate was 76% (19 students). The questionnaire consisted of factual questions covering the respondents’ demographic characteristics and attitudinal questions exploring students’ attitudes and opinions towards the CALAT TE course. The attitudinal questions were divided into closed-ended and open-ended questions. The closed-ended questions included three five-points Likert scales asking students about their Engagement and Involvement in the course (6 items), their views towards the Course Structure & Material (12 items), and the Instructor (10 items). The five-points Likert scales ranged from Strongly Disagree (1) to Strongly Agree (5). No negatively worded items were included in the scales. The open-ended questions covered four aspects: 1) course strengths; 2) course aspects most helpful to learning; 3) course aspects that constitute obstacles four learning; and 4) feedback for module improvement. This study discusses the findings of this evaluation.

Data Collection Procedure

The data were collected between May and June of the academic year 2018-2019. The CALAT module students completed the online questionnaire anonymously after their consent form (see Appendix 2 ) was signed and returned to the researcher. Said document informed the students of the research aim, its voluntary nature, a description of the research benefits to future students of the course and their language practice, a statement of confidentiality, and contact details in case of questions. It also included the identity of the researcher and institution ( Creswell, 2009 ).

Data analysis

The questionnaire’s validity and reliability were based on the steps followed in similar questionnaires used at the end of each year for the evaluation of the module, which were based on literature on online questionnaires, checked by experts, and piloted ( Creswell, 2009 ). The quantitative data were analyzed with SPSS 26. Descriptive statistics, the range, the mean, and the standard deviation (SD) were examined to determine the students’ level of satisfaction with the CALAT TE they received. The qualitative data from the open-ended questions were analyzed thematically to explore students’ views towards the course in terms of its strengths, its helpfulness, and its obstacles for learning. Students’ recommendations for improving the course were also analyzed.