TAMMY: Usability and effectiveness of a translation practice chatbot for Japanese EFL learners

Abstract

This study evaluates the usability and effectiveness of an LLM-powered chatbot, TAMMY (Translation Assistant for MMasterY), designed to support EFL translation practice for Japanese students. Tammy was designed to provide Japanese to English translation tasks, offering feedback and guidance to learners. Using chatlog data and a questionnaire, we assessed Tammy’s response validity, its success in guiding learners to accurate translations, changes in English proficiency, and student perceptions of ease of use, usefulness, and friendliness. Findings revealed high chatbot response validity (97% valid) and moderate success in guiding learners to accurate translations (64%). Students perceived Tammy as friendly and easy to use, yet their intention for continued use was neutral. While no significant relationship between Tammy usage and changes in English proficiency was found, our analysis provides insights into the design of AI-integrated translation tools. The results underscore that AI-powered chatbots can serve as effective, non-judgmental assistants for unlimited language practice, but they are most impactful as a complementary tool within a broader pedagogical framework, not a standalone solution. This work offers concrete recommendations for future chatbot design improvements to better align AI capabilities with specific learning objectives, thereby enhancing pedagogical effectiveness and promoting sustained student engagement.