EVALUATING THE VALIDITY OF MEARA AND ROGERS’ VOCABULARY APTITUDE TEST: A RASCH MEASUREMENT MODEL ANALYSIS
Abstract
Language aptitude tests, such as the LLAMA_ B3, developed by Meara and Rogers (2022), may play an important role in vocabulary learning research. Following Bokander & Bylund's (2019) advice, the construct validity of this test is examined here. Data collected from 314 participants were analyzed using Rasch analysis. All the eigenvalues fall below 2, supporting the assumption of uni-dimensionality. Q3 findings help us claim that the assumption of local independence has been met. This study found a .96 and a .84 reliability for the items and the participants, respectively. There was a convergent estimate of the results (Variance-Covariance Matrix: +2.002 and -2 log likelihood=+11.458.300). Accordingly, the test is valid for the present sample of students. As far as the analysis shows, the items represent a single underlying construct, meaning that the LLAMA_ B3 is fundamentally coherent. Psychometrically, a single trait seems to have been identified. It can be concluded that both the items and participants behaved predictably, indicating that the test is certainly worth further investigation and refinement.
References
Abrahamsson, N., & Hyltenstam, K. (2008). The robustness of aptitude effects in near-native second language acquisition. Studies in Second Language Acquisition, 30, 481-509. https://doi.org/10.1017/S027226310808073X
Alqahtani, M. (2015). The importance of vocabulary in language learning and how to be taught. International Journal of Teaching and Education, III (3), 21-34. https://doi.org/10.20472/te.2015.3.3.002.
Ameringer, V., Green, L., Leisser, D., & Turker, S. (2018). Introduction: Towards an interdisciplinary understanding of language aptitude. In S. M. Reitererm (Eds.), Exploring language aptitude: Views from psychology, the language sciences, and cognitive neuroscience (pp. 19-42). English Language Education, vol 16. Springer. https://doi.org/10.1007/978-3-319-91917-1_1
Artieda, G., & Mu˜noz, C. (2016). The LLAMA tests and the underlying structure of language aptitude at two levels of foreign language proficiency. Learning and Individual Differences, 50, 42-48. https://doi.org/10.1016/j.lindif.2016.06.023
Aryadoust, V., Ng, L. Y., & Sayama, H. (2020). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6-40. https://doi.org/10.1177/0265532220927487
Baldonado, A. A., Svetina, D., & Gorin, J. (2015). Using necessary information to identify item dependence in Passage-Based Reading Comprehension Tests. Applied Measurement in Education, 28(3), 202-218. https://doi.org/10.1080/08957347.2015.1042154
Birdsong, D. (2018). Plasticity, variability and age in second language acquisition and bilingualism. Frontiers in Psychology, 9, 81-98.
Bokander, L., & Bylund, E. (2020). Probing the internal validity of the LLAMA Language Aptitude Tests. Language Learning, 70(1), 11-47. https://doi.org/10.1111/lang.12368
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch Model: Fundamental measurement in the human sciences (3rd ed.). L. Erlbaum.
Carroll, J. B., & Sapon, S. M. (1959). Modern language aptitude test manual. Psychological Corporation.
Carroll, J. B. (1962). The prediction of success in intensive foreign language training. In R. Glaser (Ed.), Training research and education (pp. 87-136). University of Pittsburgh Press.
Carroll, J. B., & Sapon, S. (2002). Modern Language Aptitude Test: Manual. Second Language Testing.
Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using Item Response Theory. Journal of Educational and Behavioural Statistics, 22(3), 265-289. https://doi.org/10.2307/1165285
Dale, P. S., Harlaar, N., Haworth, C., & Plomin, R. (2010). Two by two: A twin study of second language acquisition. Psychological Science, 21(5), 635-640. https://doi.org/10.1177/0956797610368060
Dörnyei, Z., & Skehan, P. (2003). Individual differences in second language learning. In C. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (p. 589-630). Oxford, UK: Blackwell.
Doughty, C. J., & Mackey, A. (2021). Language aptitude: Multiple perspectives. Annual Review of Applied Linguistics, 41, 1-5. https://doi.org/10.1017/s0267190521000076
Duncan, P. W., Bode, R. K., Min Lai, S., Perera, S., & Glycine (2003). Rasch analysis of a new stroke-specific outcome scale: The stroke impact scale. Archives of Physical Medicine and Rehabilitation, 84(7), 950-963. https://doi.org/10.1016/s0003-9993(03)00035-2
Fan, J., & Bond, T. (2019). Applying Rasch measurement in language assessment. Quantitative Data Analysis for Language Assessment I, 83-102 https://doi.org/10.4324/9781315187815-5
Glas, C. A. (2016). Frequentist model-fit tests. Handbook of item response theory, volume two: statistical tools, 313-361. https://doi.org/10.1201/b19166-22
Grigorenko, E. L., Sternberg, R. J., & Ehrman, M. E. (2000). A theory-based approach to the measurement of foreign language learning ability: The Canal-F theory and test. Modern Language Journal, 84(3), 390-405. https://doi.org/10.1111/0026-7902.00076
Hambleton, R. K., Swaminathan, H. and Rogers, H. J. (1991). Fundamentals of
Item Response Theory. SAGE Publications
Harmon, J. M., Wood, K. D., & Kiser, K. (2009). Promoting vocabulary learning with the interactive word wall. Middle School Journal, 40(3), 58-63. https://doi.org/10.1080/00940771.2009.11495588
Huang, T., Loerts, H., & Steinkrauss, R. (2020). The impact of second- and third-language learning on language aptitude and working memory. International Journal of Bilingual Education and Bilingualism, 25(2), 522-538. https://doi.org/10.1080/13670050.2019.1703894
Granena, G. (2013). Cognitive aptitudes for second language learning and the LLAMA Language Aptitude Test. Language Learning & Language Teaching, 35, 105-130. https://doi.org/10.1075/lllt.35.04gra
Kagan, J. (2022). Aptitude Test: Definition, how it's used, types, and how to pass. Investopedia. https://www.investopedia.com/terms/a/aptitude-test.asp
Li, S. (2015). The construct validity of language aptitude. Studies in Second Language Acquisition, 38(4), 801-842. https://doi.org/10.1017/s027226311500042x
Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean. Rasch Measurement Transactions, 16(2), 878. https://www.rasch.org/rmt/rmt162.pdf
Linacre, J. M. (2004). Rasch model estimation: Further topics. Journal of Applied Measurement, 5(1), 95-110.
Linacre J. M. (2006). Rasch analysis of rank-ordered data. Journal of Applied Measurement, 7(1), 129-139.
Linacre, J. M. (2015). What do infit and outfit, mean-square and standardized mean? Rasch Measurement, Trans 2002, 16, 878-882.
Linacre, J. M. (2019). A user’s guide to WINSTEPS® Rasch-Model Computer Programs: Program Manual 4.4. 6. Mesa-Press, Chicago, IL. https://www.winsteps.com/winman/copyright.htm
Linck, J. A., Hughes, M. M., Campbell, S. G., Silbert, N. H., Tare, M., Jackson, S. R., Doughty, C. J. (2013). Hi-LAB: A new measure of aptitude for high level language proficiency. Language Learning, 63, 530-566. https://doi.org/10.1111/lang.12011
Loken, E., & Rulison, K. L. (2010). Estimation of a four-parameter item response theory model. British Journal of Mathematical and Statistical Psychology, 63(3), 509-525. https://doi.org/10.1348/000711009x474502
McNamara, T. F. (1996). Measuring second language performance. Longman.
Meara, P. (2005). LLAMA Language aptitude tests. Lognostics.
Meara, P. M., & Rogers, V. E. (2019). The LLAMA tests. Lognostics. https://www.lognostics.co.uk/tools/LLAMA_3/index.htm
Messick, S. (1985). Standards for educational and psychological testing. American Psychological Association.
Meyer, P. J. (2014). Applied measurement with jMetrik (1st ed.). Routledge.
Mikawa, M., & De Jong, N. H. (2021). Language neutrality of the LLAMA test explored: The case of agglutinative languages and multiple writing systems. Journal of the European Second Language Association, 5(1), 87-100. https://doi.org/10.22599/jesla.71
Nicklin, C., & Vitta, J. P. (2022). Assessing Rasch measurement estimation methods across R packages with yes/no vocabulary test data. Language Testing, 39(4), 513-540. https://doi.org/10.1177/02655322211066822
O’Malley, J. M., Parry, T. S., & Stansfield, C. W. (1993). Language Aptitude Reconsidered. The Modern Language Journal, 77(2), 1-24. https://doi.org/10.2307/328950
Pallant, J. F. (2011). SPSS survival manual: A step by step guide to data analysis using the SPSS program. Allen & Unwin.
Pimsleur, P. (1966). The Pimsleur language aptitude battery. Harcourt, Brace, Jovanovic.
Planchon, A., & Ellis, E. (2012). A diplomatic advantage? The effects of bilingualism and formal language training on language aptitude amongst Australian diplomatic officers. Language Awareness, 23(3), 203-219. https://doi.org/10.1080/09658416.2012.742907
Reiterer, S. M. (2018). Exploring language aptitude: Views from psychology, the language sciences, and cognitive neuroscience. Springer-Nature.
Robinson, P. (2002). Effects of individual differences in intelligence, aptitude, and working memory on adult incidental SLA: A replication and extension of Reber, Walkenfield and Hernstad (1991). In P. Robinson (Ed.), Individual differences and instructed language learning (pp. 211-265). John Benjamins Publishing.
Robinson, P. (2005). Aptitude and second language acquisition. Annual Review of Applied Linguistics, 25, 46-73. https://doi.org/10.1017/s0267190505000036
Rulison, K. L., & Loken, E. (2009). I’ve fallen and I can’t get up: Can high-ability students recover from early mistakes in CAT? Applied Psychological Measurement, 33(2), 83-101. https://doi.org/10.1177/0146621608324023
Sachs, R., Akiyama, Y., & Nakatsukasa, K. (2019). The value of introspective measures in aptitude-treatment interaction research. Journal of Second Language Studies, 2(2), 336-364. https://doi.org/10.1075/jsls.19001.sac
Sáfár, A., & Kormos, J. (2008). Revisiting problems with foreign language aptitude. IRAL - International Review of Applied Linguistics in Language Teaching, 46, 113-136. https://doi.org/10.1515/iral.2008.005
Singleton, D. (2017). Language aptitude: Desirable trait or acquirable attribute? Studies in Second Language Learning and Teaching, 7(1), 89-103. https://doi.org/10.14746/ssllt.2017.7.1.5
Skehan, P. (2016). Foreign language aptitude, acquisitional sequences, and psycholinguistic processes. In G. Granena, D. O. Jackson, & Y. Yilmaz (Eds.), Cognitive individual differences in L2 processing and acquisition (pp. 105-130). John Benjamins.
Sparks, R. L., Javorsky, J., & Ganschow, L. (2005). Should the modern language aptitude test be used to determine course substitutions for and waivers of the foreign language requirement? Foreign Language Annals, 38(2), 201-210. https://doi.org/10.1111/j.1944-9720.2005.tb02485.x
Stansfield, C. W., & Reed, D. J. (2004). The story behind the modern language aptitude test: An interview with John B. Carroll (1916–2003). Language Assessment Quarterly: An International Journal, 1(1), 43-56. https://doi.org/10.1207/s15434311laq0101_4
The LLAMA tests. (2020). The LLAMA Tests. https://www.lognostics.co.uk/tools/LLAMA_3/index.htm
Vinkhuyzen, A. a. E., Van Der Sluis, S., Posthuma, D., & Boomsma, D. I. (2009). The heritability of aptitude and exceptional talent across different domains in adolescents and young adults. Behaviour Genetics, 39(4), 380-392. https://doi.org/10.1007/s10519-009-9260-5
Walters, J. (2004). Teaching the use of context to infer meaning: a longitudinal survey of L1 and L2 vocabulary research. Language Teaching, 37(4), 243-252. https://doi.org/10.1017/s0261444805002491
Wen, Z. E. (2019). Working Memory as Language Aptitude [Review of Working Memory as Language Aptitude]. In Z. E. Wen (Ed.), The Phonological/Executive Model (pp. 187-214). Routledge. https://doi.org/10.4324/9781315122021-10
Wilson, M. (2011). Some notes on the term: "Wright Map". Rasch Measurement Transactions, 25(3), 1331.
Wright, B. D., Linacre, J. M., Gustafson, J. E., & Martin-Löf, P. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370. https://www.sciepub.com/reference/117986
Yan, Z., Heene, M., & Bond, T. (2020). Applying the Rasch model (4th ed.). Routledge.
Copyright (c) 2023 UNIMAS Publisher
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright Transfer Statement for Journal
1) In signing this statement, the author(s) grant UNIMAS Publisher an exclusive license to publish their original research papers. The author(s) also grant UNIMAS Publisher permission to reproduce, recreate, translate, extract or summarize, and to distribute and display in any forms, formats, and media. The author(s) can reuse their papers in their future printed work without first requiring permission from UNIMAS Publisher, provided that the author(s) acknowledge and reference publication in the Journal.
2) For open access articles, the author(s) agree that their articles published under UNIMAS Publisher are distributed under the terms of the CC-BY-NC-SA (Creative Commons Attribution-Non Commercial-Share Alike 4.0 International License) which permits unrestricted use, distribution, and reproduction in any medium, for non-commercial purposes, provided the original work of the author(s) is properly cited.
3) For subscription articles, the author(s) agree that UNIMAS Publisher holds copyright, or an exclusive license to publish. Readers or users may view, download, print, and copy the content, for academic purposes, subject to the following conditions of use: (a) any reuse of materials is subject to permission from UNIMAS Publisher; (b) archived materials may only be used for academic research; (c) archived materials may not be used for commercial purposes, which include but not limited to monetary compensation by means of sale, resale, license, transfer of copyright, loan, etc.; and (d) archived materials may not be re-published in any part, either in print or online.
4) The author(s) is/are responsible to ensure his or her or their submitted work is original and does not infringe any existing copyright, trademark, patent, statutory right, or propriety right of others. Corresponding author(s) has (have) obtained permission from all co-authors prior to submission to the journal. Upon submission of the manuscript, the author(s) agree that no similar work has been or will be submitted or published elsewhere in any language. If submitted manuscript includes materials from others, the authors have obtained the permission from the copyright owners.
5) In signing this statement, the author(s) declare(s) that the researches in which they have conducted are in compliance with the current laws of the respective country and UNIMAS Journal Publication Ethics Policy. Any experimentation or research involving human or the use of animal samples must obtain approval from Human or Animal Ethics Committee in their respective institutions. The author(s) agree and understand that UNIMAS Publisher is not responsible for any compensational claims or failure caused by the author(s) in fulfilling the above-mentioned requirements. The author(s) must accept the responsibility for releasing their materials upon request by Chief Editor or UNIMAS Publisher.
6) The author(s) should have participated sufficiently in the work and ensured the appropriateness of the content of the article. The author(s) should also agree that he or she has no commercial attachments (e.g. patent or license arrangement, equity interest, consultancies, etc.) that might pose any conflict of interest with the submitted manuscript. The author(s) also agree to make any relevant materials and data available upon request by the editor or UNIMAS Publisher.
