Speaker
Description
Theoretical Background:
Recent research has examined the relation between teacher motivational messages, teacher motivation, and student learning, primarily using student self-reports (Putwain & von der Embse, 2018; Symes & Putwain, 2016). This approach is problematic due to the susceptibility to biases in research findings (Howard, 1980; Rosenman et al., 2011). Alternative methods, such as manual classification of teacher messages, are resource-intensive (Macanovic & Przepiorka, 2024). This study proposes an AI-based approach to objectively assess teacher motivational messages, using a fine-tuned Large Language Model (LLM) to overcome these limitations.
Research Question:
This study investigates whether AI-based methods, specifically LLMs, can reliably assess pre-service teachers’ motivational messages.
Method:
We analyzed video data from 122 pre-service teachers during a 15-week course. First, the videos were transcribed with an AI software. Second, we developed a coding system for motivational messages based on self-determination theory (Ahmadi et al, 2023; Deci & Ryan, 2002) and annotated the transcripts with two human raters (κ = .72). Third, we used these annotations to fine-tune the LLM Gemma-7B (Banks & Warkentin, 2024). We split the annotated examples into a training and evaluation data set. Next, a prompt-based fine-tuning approach was applied to adapt the LLM to assess teachers' message behavior (Schick & Schütze, 2020). Finally, we evaluated the model’s predictions by comparing them to human rater annotations.
Results and Significance:
The fine-tuned LLM achieved an average accuracy of 72.27% across all categories. It performed well in classifying supportive messages but did not reach acceptable F1-score for thwarting messages and out-of-domain data due to unequal data distribution. The results indicate that while the LLM can reliably classify teacher messages, further fine-tuning with a balanced data set is needed to improve accuracy for less frequent message types.
References
Ahmadi, A., Noetel, M., Parker, P., Ryan, R. M., Ntoumanis, N., Reeve, J., Beauchamp, M., Dicke, T., Yeung, A., Ahmadi, M., Bartholomew, K., Chiu, T. K. F., Curran, T., Erturan, G., Flunger, B., Frederick, C., Froiland, J. M., González-Cutre, D., Haerens, L., . . . Lonsdale, C. (2023). A classification system for teachers’ motivational behaviors recommended in self-determination theory interventions. Journal of Educational Psychology, 115(8), 1158–1176. https://doi.org/10.1037/edu0000783
Banks, J., & Warkentin, T. (2024). Gemma: Introducing new state-of-the-art open models. https://blog.google/technology/developers/gemma-open-models/
Deci, E. L., & Ryan, R. M. (2002). Handbook of self-determination research. University Rochester Press.
Howard, G. S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4(1), 93–106. https://doi.org/https://doi.org/10.1177/0193841X8000400105
Macanovic, A., & Przepiorka, W. (2024). A systematic evaluation of text mining methods for short texts: Mapping individuals’ internal states from online posts. Behavior Research Methods, 56(4), 2782–2803. https://doi.org/10.3758/s13428-024-02381-9
Putwain, D. W., & von der Embse, N. P. (2018). Teachers use of fear appeals and timing reminders prior to high-stakes examinations: Pressure from above, below, and within. Social Psychology of Education, 21(5), 1001–1019. https://doi.org/10.1007/s11218-018-9448-8
Rosenman, R., Tennekoon, V., & Hill, L. G. (2011). Measuring bias in self-reported data. International Journal of Behavioural and Healthcare Research, 2(4), 320–332. https://doi.org/https://doi.org/10.1504/IJBHR.2011.043414
Schick, T., & Schütze, H. (2020). It's not just size that matters: Small language models are also few-shot learners. Proceedings of the 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 1–14. https://doi.org/10.48550/arXiv.2009.07118
Symes, W., & Putwain, D. W. (2016). The role of attainment value, academic self‐efficacy, and message frame in the appraisal of value‐promoting messages. British Journal of Educational Psychology, 86(3), 446–460. https://doi.org/10.1111/bjep.12117
