A Critical Review on Training Evaluation Models: A Search for Future Agenda
DOI:
https://doi.org/10.33736/jcshd.6336.2024Keywords:
teacher training, training transfer, training evaluation models, sequential explanatory mixed methods designAbstract
This paper delves into the intricate landscape of training transfer evaluation, exploring some models, strengths, and areas for improvement in assessing the effectiveness of training programs. This paper aims to analyse and compare training transfer evaluation models using a scoping review methodology. By examining their strengths, weaknesses and unique characteristics, the paper seeks to pinpoint opportunities for enhancement. The discussion centres around seminal works such as Kirkpatrick's Four-Level Model, Kaufman and Keller's Five Levels of Evaluation, the Phillips Return on Investment Model, Warr et al.’s Context, Input, Reaction and Outcome Model and Brinkerhoff’s Six-Stage Model, Bushnell’s input, process, output model. The paper underscores the importance of a comprehensive and adaptable approach to training transfer evaluation, emphasising the need for hybrid models that integrate strengths while addressing weaknesses. The exploration extends to measuring tools and research methods that enhance evaluation practices. The sequential explanatory mixed methods design emerges as an exemplar of a research methodology that seamlessly combines quantitative and qualitative approaches to offer a richer understanding of training transfer. As the paper concludes, it advocates for continuous research efforts to refine models, incorporate emerging technologies, and align evaluation practices with learning and organisational development dynamics. By revealing gaps in current knowledge and identifying previously unknown areas for improvement in training transfer, this paper contributes novel insights to the field of training transfer.
References
Alfonso, M., & Ramirez, R. (2021). The impact of professional learning communities on teacher professional development and student achievement: A meta-analysis. Educational Research Review, 36, 100366.
Ajzen, I., & Fishbein, M. (1977). Attitude-behaviour relations: A theoretical analysis and review of empirical research. Psychological Bulletin, 84(5), 888–918.
AL-Aufi, M. (2014). Implications of the Improvement of Teaching Quality for Professional Development (PD) of Academics at the Colleges of Applied Sciences (CASs) in the Sultanate of Oman (PhD Thesis). University of Waikato, New Zealand.
AL-Balushi, K. (2012). Why are teachers reluctant to provide professional development? In N. McBeath & N. AlKalbani (Eds.), Quality in ELT: Raising Pedagogical Standards (pp. 87–97, ELT Conference proceedings). Muscat: SQU.
AL-Balushi, K. (2015). Internet-based surveys: How can they be utilised to research in education? In Proceedings of INTED2015 Conference (pp. 0662–0670). Madrid, Spain: INTED.
AL-Ghatrifi, Y. (2016). The Professional Development of Teachers in Higher Education in Oman: A Case Study of English Teachers in the Colleges of Applied Sciences (PhD thesis). University of Reading, United Kingdom.
AL-Hakamani, F. (2011). Investigating the Collaborative Professional Practices of Omani EFL Teachers and Their Relation to the Quality of Their Reflection (MA Dissertation). Sultan Qaboos University, Muscat.
AL-Jardani, K. (2015). The Need for Bridging the Gap between General and Tertiary Education in Oman. English Linguistics Research, 4(2), 45-48.
Al Jabri, M., Silvennoinen, H., & Griffiths, D. (2018). Teachers’ professional development in Oman: Challenges, efforts and solutions. International Journal of Learning, Teaching, and Educational Research, 17(5), 82–103. https://doi.org/10.26803/ijlter.17.5.6.
Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick‘s levels of training criteria: Thirty years later. Personnel Psychology, 42, 331-342.
Antoniou, P., & Kyriakides, L. (2013). A Dynamic Integrated Approach to teacher professional development: Impact and sustainability of the effects on improving teacher behaviour and student outcomes. Teaching and Teacher Education, pp. 29, 1–12.
Avalos, B. (2011). Teacher professional development in Teaching and Teacher Education over ten years. Teaching and Teacher Education, 27, 10–20.
Antoniou, P., & Kyriakides, L. (2013). A Dynamic Integrated Approach to teacher professional development: Impact and sustainability of the effects on improving teacher behaviour and student outcomes. Teaching and Teacher Education, 29, 1–12.
Al-Omairi, A. (2021). The Reasons Behind Not Passing the Requirements of the New Teachers Program of the Specialised Institute for Professional Training of Teachers. ResearchGate/12816/0055865.
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616.
Arthur, W., Jr., Bennet, W., Edens, P.S., & Bell, S.T. (2003). Effectiveness of training in organisations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234-245.
Baldwin, T. T., & Ford, J. K. (2017). Transfer of training: The known and the unknown. The Annual Review of Organisational Psychology and Organisational Behaviour, 5, 201-225.
Bates, R. A. (2001). Public sector training participation: An empirical investigation. International Journal of Training and Development, 5(2), 136–150.
Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27, 341–347.
Barnett, B.G. (2005). Transferring learning from the classroom to the workplace: Challenges and implications for educational leadership preparation. Educational Considerations, 32(2), 6-16.
Bendall, T. C., & Sanders, K. (2017). Do Opportunities for Formal Learning Stimulate Follow-up in Formal Learning? A three-wave study. Human Resource Management, 56, 803–820.
Brewer, E. W. (2007). Toward a holistic evaluation framework: A synthesis of alternative approaches to evaluation. American Journal of Evaluation, 28(1), 64–78.
Brewer, T. (2007). Use of Phillips‘s five-level training evaluation and return-on-investment framework in the U.S. non-profit sector, doctoral dissertation—University of North Texas, Denton, TX.
Brinkerhoff, R. O. (1987). Achieving results from training. San Francisco, CA: Jossey-Bass.
Brinkerhoff, R. O. (1988). An integrated evaluation model for HRD. Training and Development Journal, 66-68.
Brinkerhoff, R. O. (2003). The success case method. San Francisco: Berrett-Koehler.
Burke, L., & Hutchins, H.M. (2008). Training transfer: An integrative literature review. Human Resource Development Review, 6(3), 263–296.
Bushnell, D. S. (1990). Input, process, output: A model for evaluating training. Training and Development Journal, 44(3), 41–43.
Cannon-Bowers, J. A., Salas, E., Tannenbaum, S. I., & Mathieu, J. E. (1995). Toward theoretically based principles of training effectiveness: a model and initial empirical investigation. Military Psychology, 7, 141-164.
Chen, J., & Hou, J. (2021). The influence of sustainability on training transfer: Evidence from China's manufacturing industry. Journal of Business Research, 131, 128-139. http://dx.doi.org/10.1016/j.jbusres.2021.02.044.
Chiaburu, D. S., Van Dam, K., Hutchins, H. M., & (2012). Social Support in the Workplace and Training Transfer: A Longitudinal Analysis. International Journal of Selection and Assessment, 18(2), 186–201.
Clemenz, C.E. (2001). Measuring Perceived Quality of Training in the Hospitality Industry. Doctoral thesis, The Faculty of the Virginia Polytechnic Institute and State University, USA.
Creswell, J. W. (2008). Educational research: Planning, conducting, and evaluating Qualitative and Quantitative research. New Jersey: Merrill Prentice Hall.
Creswell JW, Plano Clark VL (2011). Designing and conducting mixed methods research. 2nd ed. ThousandOaks, CA: Sage.
Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research. Sage Publications.
Devi, R. V., & Shaik, N. (2012). Evaluating training & development effectiveness - A measurement model. Asian Journal of Management Research, 2(1), 722-735.
Devins, D., & Smith, J. (2013). Evaluation of human resource development. Gold, J., Holden, R., Stewart, J., Iles, P., Beardwell, J (Eds). Human resource development: Theory and practice, chapter 7, pp. 175–204, Basingstoke: Palgrave Macmillan.
Flick, U. (2011). Introducing Research Methodology: A Beginner’s Guide to Doing a Research Project. London: Sage Publications.
Ford, J. K., Quinones, M. A., Sego, D. J., & Sorra, J. (1992). Factors affecting the opportunity to perform trained tasks on the job. Personnel Psychology, 45(3), 511–27.
Ford, J. K., & Kraiger, K. (1995). The application of cognitive constructs to the instructional systems model of training: implications for needs assessment design and transfer. In Cooper, C.L. and Robertson, I.T. (Eds), International review of industrial and organisational psychology, 1-48, Chichester: Wiley.
Foxon, M. (1989). Evaluation of training and development programs: A review of the literature. Australian Journal of Educational Technology, 5(2), 89–104.
Grossman, R., & Salas, E. (2011). The transfer of training: what matters. International Journal of Training and Development, 15(2), 103-120.
Hamblin, A. C. (1974). Evaluation and control of training. Maidenhead: McGraw-Hill.
Harrits, G.S. (2011). More than method?: A discussion of paradigm differences within mixed methods research. Journal of Mixed Methods Research, 5, 150-166.
Holton, E. F. III, Bates, R. A., Seyler, D. L., & Carvalho, M. B. (1997). Toward construct validation of a transfer climate instrument. Human Resource Development Quarterly, 8(2), 95–113.
Holton, E. F. III, Bates, R. A., & Ruona, W. E. (2000). Development of a generalised Learning Transfer System Inventory. Human Resource Development Quarterly, 11(4), 333–60.
Holton III, F. E. (1996). The Flawed Four-Level Evaluation Model. Human Resource Development Quarterly, 7(1), 5–21.
Huang, C., & Shen, H. (2021). The impact of teacher collaboration on student achievement: A meta-analysis. Educational Research Review, 32, 100373.
Hung, T. (2010). An empirical study of the training evaluation decision-making model to measure training outcome. Social Behaviour and Personality: An International Journal, 38, 87–102.
Jamjoom, M.M., & Al-Mudimigh, A.S. (2011). Training Evaluation: Towards an Effective ES Training. International Journal of Computer Science and Network Security, 11(1), 148–152.
Johnson, R., & Onwuegbuzie, A. (2004). Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher, 33(7), 14-26.
Joffe, H. (2012). 'Thematic analysis.' Qualitative Research Methods in Mental Health and Psychotherapy (eds D. Harper and A. R. Thompson). http://dx.doi.org/10.1002/9781119973249.ch
Kaufman, R., Keller, J., & Watkins, R. (1996). What works and what doesn‘t: Evaluation beyond Kirkpatrick. Performance + Instruction, 35(2), 8–12.
Kennedy, P. E., Chyung, S. Y., Winieck, D. J., & Brinkerhoff, R. O. (2014). Training professionals‘ usage and understanding of Kirkpatrick‘s level 3 and level 4 evaluations. International Journal of Training and Development, 18(1), 1-21.
Khalid, M., Rehman, C., & Ashraf, M. (2012). Exploring the link between Kirkpatrick (KP) and context, input, process and product (CIPP) training evaluation models and its effect on training evaluation in public organisations of Pakistan. African Journal of Business Management, 6(1), 274–279.
Kim, Y. J., Lee, H. J., & Kwon, H. J. (2020). Examining the role of teacher motivation in student learning outcomes: A cross-sectional study. Asia Pacific Education Review, 21(3), 401-413.
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco: Berrett-Koehler Publication.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of Cognitive, Skill-Based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation. Journal of Applied Psychology, 78(2), 311-328.
Lockwood, S. L. (2001). Enhancing employee development: Development and testing of a new employee orientation protocol. Doctoral thesis, California School of Professional Psychology, San Diego.
McKenna, E. F., & Beech, N. (2014). Human resource management: A concise analysis. New York: Pearson Education.
Milne D.L. (2007). Evaluation of staff development: The essential ‗SCOPPE. Journal of Mental Health, 16, 389–400.
Morse, J., & Niehaus, L. (2009). Mixed methods design: Principles and procedures. Walnut Creek, CA: Left Coast Press.
Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(1), 1–7. https://doi.org/10.1186/s12874-018-0611-x.
Nickols, F. W. (2005). Why a stakeholder approach to evaluating training. Advances in Developing Human Resources, 7(1), 121-134.
Noe, R. (2016). Employee Training and Development. Boston: Irwin/McGraw-Hill.
Passmore, J., & Velez, G. A. (2012). International coaching psychology review: A bibliography of research using coaching psychology themes by academic institutions and professional bodies. International Coaching Psychology Review, 7(1), 37-50.
Peterson, J., Pearce, P. F., Ferguson, L. A., & Langford, C. A. (2017). Understanding scoping reviews: Definition, purpose, and process. Journal of the American Association of Nurse Practitioners, 29(1), 12–16. https://doi.org/10.1002/2327-6924.12380.
Phillips, J. J. (1996). How much is the training worth? Training and Development, 50(4), 20–24.
Phillips, J. J. (2005). Investing in your company's human capital: Strategies to avoid spending too little or too much. New York: Amacom.
Preskill, H., and Russ-Eft, D. (2005). Building evaluation capacity: 72 activities for teaching and training. Thousand Oaks: Sage.
Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860.
Reio, T.G., Rocco, T.S., Smith, D.H., & Chang, E. (2017). A Critique of Kirkpatrick‘s Evaluation Model. New Horizons in Adult Education & Human Resource Development, 29(2), 35–53.
Robertson, P. J. (2004). Beyond Return on Investment (ROI) and Kirkpatrick: A guide to evaluating training. Journal of European Industrial Training, 28(2/3/4), 23–31.
Roca-Puig, V., & Llorens-Montes, F. J. (2021). How corporate sustainability affects the transfer of training: The mediating role of organisational learning capability. Journal of Business Research, 124, 154-166. https://doi.org/10.1016/j.jbusres.2021.01.022.
Russ-Eft, D. F., Preskill, H., & Sleezer, C. (1997). Human resource development review: Research and implications. Thousand Oaks, CA: Sage Publications.
Saks, A. M. and Burke, L. A. (2012). An investigation into the relationship between training evaluation and the transfer of training. International Journal of Training and Development, 16(2), 118-12.
Salas, E. & Cannon-Bowers, J. A. (2001). Reflections on shared cognition. Journal of Organisational Behaviour, 22, 195–202.
Salas, E., Tannenbaum, S. I., Kraiger, K., Smith-Jentsch, K. A., & Cannon-Bowers, J. A. (2021). The science of training and development in organisations: What matters in practice for 2021 and beyond. Psychological Science in the Public Interest, 22(2), 49-113.
Shannon-Baker, P. (2015). Making Paradigms Meaningful in Mixed Methods Research. Journal of Mixed Methods Research, 10(4), 319–334. http://dx.doi.org/10.1177/1558689815575861.
Stewart G.L. & Brown K.J. (2011). Human Resource Management Linking Strategy to Practice. Danver: John Wiley & Sons.
Smith, J., & Johnson, A. (2023). Rethinking Kirkpatrick's model: A critique and proposal for a comprehensive evaluation framework. Human Resource Development Review, 28(4), 457–473.
Stokking, K. (1996). Levels of Evaluation: Kirkpatrick, Kaufman and Keller, and Beyond. Human Resource Development Quarterly, 7(2), 179–183.
Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In Madaus, G. F., Scriven, M., and Stufflebeam, D. L. (Eds.), Evaluation models, 117–141. Boston, MA: Kluwer-Nijhoff.
Tamkin, P., Yarnall, J., and Kerrin, M. (2002). Kirkpatrick and Beyond: A review of models of training evaluation. Brighton: Institute of Employment Studies, Report 392.
Tannenbaum, S. I. and Yukl, G. (1992). Training and development in work organisations. Annual Review of Psychology, 43, 399-441.
Tashakkori, A., & Creswell, J. W. (2007). The new era of mixed methods. Journal of Mixed Methods Research, 1(1), 3-7.
Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed-methods research: Integrating quantitative and qualitative approaches in the social and behavioural sciences. Thousand Oaks, CA: Sage.
Tennant, Ch., Boonkrong, M., and Roberts, P.A.B. (2002). The design of a training programme measurement model. Journal of European Industrial Training, 26(5), 230-240.
Topno, H. (2012). Evaluation of training and development: An analysis of various models. IOSR Journal of Business and Management, 5(2), 16–22.
Tzeng, G.H., Chiang, C.H., and Li, C.W., (2007). Evaluating intertwined effects in e-learning programs: a novel hybrid MCDM model based on factor analysis and DEMATEL. Expert Systems with Applications, 32, 1028–1044.
Wang, G. G., & Wilcox, D. (2006). Training evaluation: Knowing more than is practised. Advances in Developing Human Resources, 8(4), 528–539.
Warr, P., Bird, M., & Rackham, N. (1970). Evaluation of Management Training. London: Gower Press.
Watkins, R., Leigh, D., Foshay, R., & Kaufman, R. (1998). Kirkpatrick Plus: Evaluation and Continuous Improvement with a Community Focus. Educational Technology Research & Development, 46(4), 90–96.
Wick, C. (2003). Going beyond the finish line: Keep them learning long after the training ends. Training and Development, 57(12), 17–19.
Wick, C.W., Pollock, R.V., & Jefferson, A. (2010). The six disciplines of breakthrough learning: How to turn training and development into business results. San Francisco, CA: Pfeiffer.
Whittemore, R., Chao, A., Jang, M., Minges, K. E., & Park, C. (2014). Methods for knowledge synthesis: an overview. Heart & Lung, 43(5), 453–461. https://doi.org/10.1016/j.hrtlng.2014.05.014
Yamkovenko, V. and Holton, E. F. (2010). Toward a theoretical model of dispositional influences on transfer of learning: A test of a structural model. Human Resource Development Quarterly, 21, 381- 410.
Zinovieff, M.A. and Rotem, A. (2008). Review and Analysis of Training Impact EvaluationMethods, and Proposed Measures to Support a United Nations System Fellowships Evaluation Framework., Geneva, Switzerland: WHO‘s Department of Human Resources for Health.
Downloads
Published
How to Cite
Issue
Section
License
Copyright Transfer Statement for Journal
1) In signing this statement, the author(s) grant UNIMAS Publisher an exclusive license to publish their original research papers. The author(s) also grant UNIMAS Publisher permission to reproduce, recreate, translate, extract or summarize, and to distribute and display in any forms, formats, and media. The author(s) can reuse their papers in their future printed work without first requiring permission from UNIMAS Publisher, provided that the author(s) acknowledge and reference publication in the Journal.
2) For open access articles, the author(s) agree that their articles published under UNIMAS Publisher are distributed under the terms of the CC-BY-NC-SA (Creative Commons Attribution-Non Commercial-Share Alike 4.0 International License) which permits unrestricted use, distribution, and reproduction in any medium, for non-commercial purposes, provided the original work of the author(s) is properly cited.
3) The author(s) is/are responsible to ensure his or her or their submitted work is original and does not infringe any existing copyright, trademark, patent, statutory right, or propriety right of others. Corresponding author(s) has (have) obtained permission from all co-authors prior to submission to the journal. Upon submission of the manuscript, the author(s) agree that no similar work has been or will be submitted or published elsewhere in any language. If submitted manuscript includes materials from others, the authors have obtained the permission from the copyright owners.
4) In signing this statement, the author(s) declare(s) that the researches in which they have conducted are in compliance with the current laws of the respective country and UNIMAS Journal Publication Ethics Policy. Any experimentation or research involving human or the use of animal samples must obtain approval from Human or Animal Ethics Committee in their respective institutions. The author(s) agree and understand that UNIMAS Publisher is not responsible for any compensational claims or failure caused by the author(s) in fulfilling the above-mentioned requirements. The author(s) must accept the responsibility for releasing their materials upon request by Chief Editor or UNIMAS Publisher.
5) The author(s) should have participated sufficiently in the work and ensured the appropriateness of the content of the article. The author(s) should also agree that he or she has no commercial attachments (e.g. patent or license arrangement, equity interest, consultancies, etc.) that might pose any conflict of interest with the submitted manuscript. The author(s) also agree to make any relevant materials and data available upon request by the editor or UNIMAS Publisher.
