EXAIT: Educational eXplainable Artificial Intelligent Tools for personalized learning

As artificial intelligence systems increasingly make high-stakes recommendations and decisions automatically in many facets of our lives, the use of explainable artificial intelligence to inform stakeholders about the reasons behind such systems has been gaining much attention in a wide range of fields, including education. Also, in the field of education there has been a long history of research into self-explanation, where students explain the process of their answers. This has been recognized as a beneficial intervention to promote metacognitive skills, however, there is also unexplored potential to gain insight into the problems that learners experience due to inadequate prerequisite knowledge and skills that are required, or in the process of their application to the task at hand. While this aspect of self-explanation has been of interest to teachers, there is little research into the use of such information to inform educational AI systems. In this paper, we propose a system in which both students and the AI system explain to each other their reasons behind decisions that were made, such as: self-explanation of student cognition during the answering process, and explanation of recommendations based on internal mechanizes and other abstract representations of model algorithms.


Introduction
The influence of artificial intelligence (AI) has bought about much change in various fields, from legal and finance to medicine and education. In particular, AI has been applied to decisions that rely on estimation, recommendation, and prediction that can be provided by models, and has led to ethical scrutiny of the transparency of such systems . Much of the research into explainable AI (XAI) has focused on verifying the rationale behind results from such systems to explain to experts and knowledgeable stakeholders and such a use case also exists in education when explaining to teachers, however there are also additional facets of explanations from learning systems that should be considered, such as: students that are learning using such systems need to be able to understand explanations given by such system. Duffy and Azevedo's (2015) research into prompt and feedback mechanisms in intelligent tutoring systems (ITS) shows that it can have a positive effect on a student's motivation and lead to higher achievement in self-regulated learning with the system.
Consequently, we propose that XAI in education should encompass two essential components: providing explanations to raise students' awareness of their learning progress and explaining recommendations to establish trust and motivate students to continue using the system.
While the explanation to stakeholders of decisions and recommendations made by artificial intelligence in education has recently been gaining much attention, research into self-explanation from students in the context of education has a long history, and its prominence can be traced back to Chi's work on the topic (Chi et al., 1989). The act of self-explanation involves the student reflecting on the process in which they took to produce an answer to a task, and specifically state the thought process, reasoning behind, or skills that were applied while they were completing the task. Bisra et al. (2018) conducted a meta-analysis of 64 research reports on the use of self-explanation and its effects on learning outcomes. They found that self-explanation can be a valuable intervention in various learning contexts, but acknowledged issues with instructor-guided self-explanation. The authors recommended exploring the use of system-generated selfexplanation scaffolds to improve the effectiveness of the intervention. Additionally, although Chi et al. (1989) mentioned the potential for self-explanations to inform AI-driven learning systems, this area of research is still ongoing. It is possible for such systems to learn how to explain the answer process of a question in mathematics by analyzing selfexplanations provided by high-performing students and providing those explanations as a scaffold for struggling students. Answer process analysis, such as pen stroke input time series analysis, could be used to identify weaknesses in prerequisite knowledge and inform the use of self-explanation in the context of the answer process (Yoshitake et al., 2020). Wu et al. (2021) introduced theoretical frameworks for symbiotic learning systems, in which both the learner and the system can learn from each other using reinforcement learning. While the primary objective of these platforms and frameworks is to enhance the system's performance and effectiveness by adapting the model to the learners' behavior, the learners themselves also benefit from the system's feedback and guidance. However, these systems rely only on the information provided by educational systems, such as learning behavior logs and assessment outcomes, and do not attempt to probe or query learners to gain insight into their state. There is effectively a double black box problem as shown in Figure 1, where the decisions of the AI system rely on an algorithm that could be difficult for the learner to understand, and a learner who makes decisions based on cognition and latent knowledge that the AI system needs to accurately estimate (Fischer et al., 2020).
The main problems this research aims to address are in facilitating both AI and learner explanation for not only mutual understanding but also in a form that has the potential to offer educational benefit to the learner while providing information benefit for the AI system. The ultimate goal of the project is to construct a framework to encompass possible solutions to these main problems, which can be supported by the components of the EXAIT system. While previous research has examined to a degree individual components of the system, such as: recommendation explanation and self-explanation, to the best of the author's knowledge there is no research that aims to tackle both problems as a symbiotic system.
The current paper proposes a symbiotic learning system that not only adapts to the learner's performance but also promotes mutual understanding through the use of explanations by both the learner and the system. The proposed system is called EXAIT: Educational eXplainable AI Tools, and it aims to combine the benefits of both types of explanations in education into a single learning tool that can co-evolve symbiotically through the learner's self-explanation and AI-generated explanation. Initially, the system provides AI recommendations of potential learning paths to foster trust and learner awareness. The student then completes a task, such as a mathematics question, using a stylus pen to input their working out and final answer into the system for evaluation. The system then prompts the student to self-explain their answer process by replaying it interactively and annotating points in time to indicate the knowledge applied to overcome sub-problems. Time series analysis is then applied to the self-explanation and answer process data to extract information, such as backtracking or stuck points, that could indicate problems with dependent or related knowledge. The ultimate goal of the system is Fig. 1 A double black box: AI systems and learners need to explain to each other to increase understanding and forms the basis of the EXAIT system to complete the symbiotic explanation cycle by incorporating the self-explanation analysis into the AI recommendation model.
In this paper, we propose the EXAIT system based on mutual explanation by the AI system and Learner. Case studies that have been conducted while developing individual components of the system are introduced and the initial findings are discussed to inform the construction of the EXAIT cycle.

Related work
The EXAIT cycle comprises of a number of different components from different sub-fields within educational technology, such as: self-explanation in education, educational recommender systems, symbiotic learning systems, and explainable AI in education. While self-explanation and educational recommender systems have a rich past in the field of education, symbiotic learning systems, and explainable AI in education have recently been gaining attention as important areas of investigation.

Symbiotic learning systems
Earlier studies have introduced theoretical frameworks for symbiotic learning systems (Wu et al., 2021), in which the learner learns from the system, and the system also learns from the learner through reinforcement learning. Walsh et al. (2017) proposed a highly conceptual symbiotic learning system for supporting the delivery of online learning. They focused on adapting the system based on previous use to optimize the learning experience from the perspective of cognitive learning and learner affect based on a number of costs and reword measures, such as: the time taken to design, and the time spent and effectiveness of learning by students. While the primary objective of such platforms and frameworks is to enhance the system's performance and effectiveness by adapting the model to the learners' behavior, the proposed symbiotic learning system in this paper aims to additionally provide mutualistic symbiosis. It focuses not only on improving the system's performance but also on promoting mutual understanding through the use of explanations by both the learner and the system.
Other research has investigated the use of different modalities of symbiotic learning systems. The EU EASEL project (Reidsma et al., 2016) explores the potential impact and relevance of robots in educational settings with a focus on creating symbiotic social robots that make transformative contributions in the classroom. The key aim is to develop a Synthetic Tutoring Assistant that incorporates features of human tutors through learning models and adaptive strategies to support students in their learning tasks. The importance of social interaction and affective engagement between students and robots in the learning process is emphasized as an important feature of the symbiotic social robot. In contrast to the EASEL project, EXAIT focuses less on social and affective engagement and more on a learning cycle that benefits both the learner and system to enhance each other's progress.
Work on virtual reality-based ubiquitous symbiotic learning systems and their effect on learning outcomes has also been explored (Zhang et al., 2022). It was suggested that due to the encompassing nature of virtual reality systems, the symbiotic learning system would have greater access to data about the learner in the virtual ubiquitous environment. While it may be advantageous to use such environments due to their ability to collect data effectively it is important not to inhibit learning for the sake of data collection, and therefore the EXAIT system aims to engage students through self-explanation which has been shown to be an effective learning intervention (Bisra et al., 2018).
Other work into symbiotic learning systems has examined their ability to facilitate knowledge transfer. Kinsner and Saracco (2019) highlight the potential of symbiotic digital twin-based learning systems to revolutionize education in a rapidly changing world. The proposed digital twins facilitate personalized learning and proactive education programs to help individuals keep up-to-date and continuously update their skills throughout their lives by considering not only their body of knowledge (BoK) but also their body of experience (BoE). Winne (2021) proposes incorporating trace data about learning processes in open learner models (OLMs), enabling the generation of learning analytics that inform selfregulating learners about productive learning adaptations. By embracing self-regulated learners and leveraging big data , OLMs can collaborate symbiotically with learners, improving learner models and enhancing self-regulated learning experiences. Eikeland (2013) discusses the role of symbiotic learning systems in the transfer of knowledge, drawing inspiration from the philosophies of Plato and Aristotle who emphasized the importance of intellectual commons and communities for learning. It is argued that these systems could serve as a knowledge conduit across a learner's life and to different learners. The EXAIT system currently partially embodies these aspects in that learners are self-explaining to the system, which enables it to better understand their learning state.

Self-explanation in education
Many works have suggested that self-explanations offer learning benefits (Bisra et al., 2018), and are often combined or are an integral part of other instructional designs, such as worked examples and contrasting case instruction (Sidney et al., 2015). However, it has been noted that it is still unknown if the use of self-explanation is the most effective intervention and use of a learner's time (McEldoon et al., 2013;Rittle-Johnson et al., 2017).
In mathematics (Hänze & Leiss, 2022;Renkl, 2017) and other science subjects, such as chemistry (Crippen & Earl, 2007), worked examples are used to provide students with not only solutions, but also explanations from an expert mental model that are approachable from a novice mental model (Sweller, 2006), and when coupled with self-explanations provide a prompt for deeper understanding (Renkl, 2005). Crippen and Earl (2007) proposed a tool with which students can perform worked examples and self-explanation for a chemistry course. By conducting a quasi-experiment, they found that by combining worked examples with self-explanation prompts there was an improvement over just worked examples in the case of performance, problem solving skills, and self-efficacy. Tajika et al. (2007), also confirmed that self-explanation supports effective learning and that students with high self-explanation capability generated more self-explanations showing a deeper understanding of the questions that those who had low self-explanation.
While research into self-explanation has mainly been focused on STEM subjects, recent working has also explored its application in other fields, such as language learning and reading comprehension. The iSTART tutoring system was developed to assist students in paraphrasing during reading comprehension by analyzing their self-explanation within the context of the target text. It utilized a combination of NLP methods for self-explanation analysis, scaffolded instructions, and self-explanation practice to encourage the development of necessary skills (McNamara et al., 2007). An automated evaluation algorithm was used to assign scores to students' self-explanations based on the relevance of topics to the target text, with high scores given to those that demonstrated strong alignment and lower scores to those that were off-topic or short in length. The system was found to be effective in supporting selfexplanation across various fields of text (Jackson et al., 2010).
In contrast, the EXAIT system aims to use self-explanation analysis to inform learner models about their strengths and weaknesses, rather than relying solely on automated evaluation of the content. The analysis of self-explanation is intended to influence the recommendation of learning paths for the student, thereby closing the gap in the explanation co-evolution cycle.

Explainable AI in education
Recently XAI has begun to attract attention in the field of education for emerging concerns about Fairness, Accountability, Transparency, and Ethics (Khosravi et al., 2022). XAI is one of the emerging methods for increasing trust in AI systems, which promotes the use of methods that "enable a human user to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners" (Gunning, 2017).
Interpreting the decision process of the model and thereby providing explanations can be expected to have a positive impact on students' academic performance by improving their sense of conviction and increasing their confidence in the AI. For the teacher, by presenting an explanation of why the question was recommended to improve academic performance, it shows what explanatory methods are useful for improving academic performance.
Artificial intelligence in education has enabled the development of e-learning systems that simulate student's knowledge and experience to provide personalized support to students (Nwana, 1990;Self, 1974;Wenger, 2014). AI-supported e-learning refers to the use of AI techniques (e.g., Fuzzy Logic, Decision tree Bayesian networks, Neural Networks, Genetic algorithms and hidden Markov models) in e-learning (i.e., using computer and network technologies for learning or training) (Colchester et al., 2017).
Recent meta-review reported that the most identified AI-supported e-learning systems were Adaptive Learning Systems, the second identified kind of AI-enabled learning system is intelligent tutoring system and Recommendation system is the last one (Kabudi et al., 2021).
Therefore AI-supported recommendation systems in the educational field were not well studied.
On the other hand, recommender systems are everywhere in our lives, and AI is being used here as well. For example, Amazon recommends products with Collaborative Filtering (Smith & Linden, 2017) and Netflix recommends movies using deep learning (Amatriain & Basilico, 2015). In this research field of recommendation mainly in e-commerce, explainable recommendations, which provide explanations about why an item is recommended, have received much attention for improving transparency, persuasiveness and trustworthiness (Zhang & Chen, 2020). Based on these studies, also in education, we suppose explanations from a learning system provide additional benefits for students learning from the explanations given by the system. Previous research on intelligent tutoring systems has shown that student motivation in system-based selfregulated learning can be improved by prompting and feedback mechanisms, leading to higher achievement (Duffy & Azevedo, 2015). Recent adaptive learning systems have attempted to recommend learning materials based on complex methods such as deep learning methods  and reinforcement learning methods . However, the mechanism and output of these methods are difficult to interpret, which may lead to the decrease of students' beliefs that they are able to do the task and their perceived values of completing the task, which further decreases their motivation to participate (Wigfield & Eccles, 2000). Further, eXplainable AI has begun to attract attention in the field of education for emerging concerns about Fairness, Accountability, Transparency, and Ethics (Khosravi et al., 2022). Explanations interpreting the decisionmaking process of AI are very important for teachers because they must be accountable to students, parents or the governments. Teachers need to know why such feedback was given by the AI, and interpreting why it was given may help teachers improve their teaching skills.
The field of education technology has a long history of applying AI to support effective learning, with ITS being the forefront of the field for a long time. Systems such as MATHia (Ritter & Fancsali, 2016), ASSISTments (Heffernan & Heffernan, 2014) or ALEKS (Cosyn et al., 2021) are widely used within K-12 mathematics education to improve students' learning outcomes. These systems provide individualized step-by-step scaffolding and feedback for each student based on learners comprehension (Alkhatlan & Kalita, 2018). ITS has been evaluated for its learning gains compared to standard educational methods such as simultaneous instruction in a classroom or one-on-one education by humans (Kulik & Fletcher, 2016).
Some explainable recommendation research has been carried out in the field of education: Wikipedia recommendation in learning textbooks (Rahdari et al., 2020), recommendation in programming classes (Barria-Pineda et al., 2021) (both for higher education), and cognitive training for primary or secondary school children (Tsiakas et al., 2020). And various methods of explaining recommendations have been proposed using two different approaches, model-intrinsic and post-hoc approaches. In model-intrinsic, rule-based (Conati et al., 2021), keyword-based , and concept-based  were proposed to generate explanations. Takami et al. (2021  There have also been a variety of studies on mathematics subjects. Preschools math (Gulz et al., 2020), mathematical instruction , metacognitive scaffolding for learning by teachable agent (Matsuda et al., 2020), teachable agent in chat system (Tärning et al., 2019) and personalizing algebra to students' individual interests in an intelligent tutoring system: Moderators of impact (Walkington & Bernacki, 2019), modeling and predicting the active video-viewing time in a large-scale E-learning system (Beck & Chang, 2007). While there have been many AI-supported learning studies in mathematics, there have been few studies that recommend quizzes by AI and provide further reasons for the recommendation. So we addressed mathematics learning as an AI-supported research using Bayesian knowledge tracing model parameter based Explainable Recommender System for middle school students.
In the context of K-12 math education, identifying the quizzes of the appropriate difficulty is essential to improving the students' understanding of math concepts. Previous knowledge tracing works have focused on estimating the knowledge states of the students and predicting students' performance of the learning materials (Corbett & Anderson, 1994;Huang et al., 2021;Liu et al., 2019;Nakagawa et al., 2019;Sun et al., 2021). This is based on an assumption that learning happens when students attempt tasks in the zone of proximal development (Vygotsky, 1978), that is, the tasks they cannot achieve by themselves but can achieve with assistance. However, these works did not consider how knowledge states are improved and what improvement brought by the "proximal" learning materials should be prioritized. In the EXAIT framework, we propose a model-driven quiz recommender system which not only considers the difficulty of the quiz but also the expected learning outcome of solving that quiz. Among various learning outcomes, we focus on the average improvement of the understanding of related math concepts. By doing so, a quiz that helps the student to practice weaker concepts will be prioritized in the recommendation.

Methods for generating explanations of recommendations and decisions from AI in education
Recommendation explanations can be generated from different data sources and provided in different display styles (Tintarev & Masthoff, 2015), i.e., a relevant user or item, a sentence, an image or a set of reasoning rules. Basically, there are two approaches to generate explanations in recommender systems; model-intrinsic and post-hoc (Zhang & Chen, 2020). In the model-intrinsic approach, the models' mechanism is transparent and the explanation explains exactly how the model generates a recommendation. To this end, the processes of generating recommendation and generating explanation are mutually dependent . Though in this model-intrinsic approach, the goal of being explainable sometimes can constrain the model from being complex and "deep". For example, deep learning based knowledge tracing, represented as deep knowledge tracing (DKT) (Piech et al., 2015), to model the knowledge state using recurrent neural network and other side information achieved better prediction accuracy compared to ordinary Bayesian knowledge tracing based approaches (Su et al., 2018;Yeung & Yeung, 2018). Deep learning based approach achieved state-of-the-art accuracy in knowledge state prediction, but it models the relation between the sequential learning activities and the knowledge state implicitly, so it is difficult to interpret the decision process in the model.
In contrast, the post-hoc approach generates the explanation after a recommendation is generated (i.e., Providing simple statistical information like "70% of your friends bought this item"), but the explanations by post-hoc does not mean that they are fake, they are just decoupled from the model. As a result, the model is allowed to be a "black box" and the explanation does not necessarily explain why an item is recommended based on the recommender model. In the educational research context, several methods for generating explanations were proposed.

Educational eXplainable Artificial Intelligent Tools (EXAIT)
The EXAIT system  is built upon the LEAF framework that has been developed to support the distribution of learning materials, collection, and automated analysis of learning behavior logs in an open and standards-based approach . The main components of the framework are shown in Figure 2 and consist of: an LMS, such as Moodle which acts as a main system from which the different components of the EXAIT system can be accessed from various courses; the BookRoll reading system where learning materials can be read and associated quizzes can be answered; the main centralized learning record store (LRS) for collecting learning behavior logs from all of the different tools; and a learning analytics dashboard LogPalette where results and recommendations given by the system can be viewed by teachers and students. Components of the system are interconnected by LTI to allow the seamless authentication of teachers and students between the tools. Data from all of the components are sent to the LRS in xAPI format. A main criterion for developing the EXAIT systems is to provide recommendations, explanations, and self-explanation analysis and feedback as close to real-time as possible. The motivation behind this is that students expect the system to adapt to match their current learning state, and therefore should recommend another question based on their actions to date. Also, the system should be easy to integrate many off-theshelf AI algorithms with ease. As LogPalette currently does not process data fast enough, the data in the LRS is processed in real-time by a dedicated data importer for EXAIT built using the Mongodb change streams pipeline in Python to a SQL database that acts as a cache for processed data. This enables the EXAIT recommender to provide recommendations on the latest data and instantly update based on learner actions while using the system. Recommendations are provided through the interface of LogPalette, which is generated by an internal API from the EXAIT recommender system built in Python. By implementing the EXAIT recommender in Python, it enables easy application of AI systems from a wide range of pre-existing tools available in the Python community, such as the PyBKT library (Badrinath et al., 2021) that is used as a base on which recommendation customizations and explanations can be augmented.
The EXAIT system is based on a symbiotic cycle between the learner and the AI engine as shown in Figure 3 that consists of the following steps: firstly, the learner reads learning materials and answers quizzes using the BookRoll system. After answering a quiz, the learner self-evaluates the correctness of his/her answer by referring to the standard solution. Then, the learner reviews his/her answer and explains why he/she conducted those steps in the answer using the answer process analysis and self-explanation tool in LogPalette. The AI engine collects learners' answers, self-evaluations, and self-explanations during these processes. Then the AI engine analyzes the self-explanations by the learner to identify potential stuck points in the answering process and link them to relevant related concepts.
This information is then combined with the learner's quiz answer history and analyzed to generate a limited number of appropriate recommendations for the learner based on their current situation. Finally, the AI engine explains why the recommendations have been made. The learner can select a quiz or learning material that they deem as being appropriate based on their interpretation of the explanation and what they perceive as meeting their current learning needs. Figure 4 provides an overview of the EXAIT system, where the LEAF framework serves as the foundation. The system comprises of an abstracted model layer consisting of two main aspects: data-driven and model-driven, denoted as ③ and ④ respectively. The model-driven aspect features two key models: the knowledge model, which is a knowledge graph of course concepts, and the student model, which monitors learning progress by analyzing data collected in the LEAF platform. The data-driven aspect includes an evidence model that guides the system based on past interventions. At the top of the diagram is the interaction layer, which demonstrates the symbiotic nature of the system during student learning. Two critical aspects of the interaction are shown: the system's capacity to explain recommendations or decisions made to stakeholders denoted as ①, and the reciprocal self-explanation by students of their answers submitted while using EXAIT denoted as ②. This reciprocal explanation between the system and students forms the basis of the symbiotic process, which is the core of the EXAIT system. The current evaluation of the proposed system is focused on English and Mathematics education in Japanese secondary schools. This decision was influenced by the Japanese government's GIGA-school program, which aims to provide one computer per student to all children in compulsory education, with a primary focus on secondary schools.
Additionally, Brod (2021) suggests that self-explanation is more effective when used in secondary schools or higher, further justifying the selection of secondary schools as a suitable target for the EXAIT system. In this paper, we will outline the implementation of the EXAIT system specifically for Mathematics and English education.

Case studies
The EXAIT system was implemented at public schools in Japan. Teachers and students in Mathematics and English subjects used the system extensively in and outside their daily classes. A list of the case studies introduced in this article are shown in Table 1.

Determining stuck points in math problem-solving and answer self-explanation
In some circumstances, handwriting input still is an important part of the learning process   Takii et al. (2020Takii et al. ( , 2021aTakii et al. ( , 2021b for students even in digitized education environments. For example, while there have been advances in input methods, particularly in mathematics, it is advantageous for young students to use intuitive pen-based input. BookRoll supports the input of handwritten memos and answers to quizzes as shown on the left in Figure 5, and the pen stroke data is collected as time series vectors which are stored in the LRS in xAPI format. To support teachers in classes utilizing this input process, a pen stroke analysis module was developed for the LogPalette dashboard as shown in the center and right in Figure 5 (Yoshitake et al., 2020). Usually, when examining a handwritten answer on paper, a teacher can try to determine the points in the answering process where the student struggled or had difficulty.
However, the teacher cannot know how long a student took to finish different sections of the answer process or parts that were erased and revised, both of which can be indications of sections in which the learner had some difficulty or stuck point. By analyzing the pen stroke data from students' answers, the time between strokes can be visualized to show potential stuck points that would otherwise be impossible to detect from paper-based answers. Additionally, teachers often source worked examples for the answers of students in mathematics classes, and this usually involves the time-consuming process of reviewing all of the previous answers. To support this process, a clustering function and example visualization was developed to enable the quick selection of ideal examples for an explanation. An experiment was conducted over three months with 360 students to determine the effectiveness of the system from the perspective of two teachers. An interview was conducted after the experiment period and both teachers highly rated the system for identifying at-risk students, reviewing students' answers, detecting stuck points from the visualization of delayed stroke analysis, and understanding typical answers from high and low performing students. The teachers also expressed a high desire to continue using the system in their classes.
The system includes a feature where students are prompted to explain their handwritten answers to recommended questions. A current user interface, displayed in Figure 6, includes a time delay analysis of the pen strokes in the answer, indicating where students paused during the process (Yoshitake et al., 2020). Students can playback and review their answers using the ① handwritten answer playback interface at the top, adjusting the playback rate and jumping to different parts of the process using the jump bar. If playback is paused, students can create a new explanation for a specific step in the answer process. During playback, the ② self-explanations of answers are shown on the right of the screen, scrolling through each explanation and highlighting the current one, as seen in Figure 6.
Research conducted by Chiu and Chi (2014) has demonstrated that self-explanations can enhance meta-cognitive skill utilization in students by prompting them to analyze their answers and draw on previously acquired knowledge to solve sub-problems. Furthermore, the act of generating and completing self-explanations can increase students' awareness of their own learning and facilitate the conveyance of new explanations for acquired knowledge using examples, resulting in more sophisticated knowledge construction.
However, despite these potential benefits, it was observed while collaborating with class teachers that most students do not perceive this task as valuable and view it as a hindrance. Additionally, students who have not grasped the necessary knowledge struggle with producing explanations in comparison to high-achieving students.
Students who lack the necessary knowledge to create self-explanations for their answers may encounter difficulties and produce subpar examples (Chi, 2000). Consequently, providing support to such students may be necessary. One possible approach to providing such support is by employing the method of self-explanation, which can range from open to limited, as suggested by Wylie and Chi (2014). The available options include openended self-explanation (natural language), focused self-explanation, scaffolded selfexplanation, glossary/resource-based self-explanation, and guided self-explanation.
Currently, the EXAIT system utilizes an open-ended self-explanation interface that allows students a high degree of freedom of expression. However, this approach may not be reliable when the instructor is not present to guide the students through the process, as noted in previous research by Bisra et al. (2018). In the next iteration of the system, we plan to incorporate a guided process map-based self-explanation interface. This interface will provide students with predetermined keywords generated by the system based on the required knowledge to complete the task. The students will be prompted to arrange the keywords in order of the answer process and assign appropriate points in time when they utilized the knowledge.

Answer self-explanation in mathematics
First, we proposed a Rubric-Based Model of knowledge that defines the knowledge elements needed to solve the content as described in Table 2 (Nakamoto et al., 2021). For the subject of an eighth-grade math problem on linear functions, those knowledge elements and their corresponding model answer self-explanatory sentences are shown in Table 3.  Can-do descriptors that clearly describe all the essential knowledge components of the quiz and are used to create labels and sample selfexplanations for scoring.
Sample answer (of self-explanations) Model answers of self-explanations with knowledge components, which are prepared according to the step rubric number. Table 3 Rubrics and a sample answer of self-explanation in a quiz

Number Rubric
Sample answer of self-explanations Step 1 Be able to find the equation of a linear function from two points.
Substituting the y-coordinate of p into the equation of the line AC.
Step 2 Be able to find the equation of the line that bisects the area of a triangle.
Find the area of triangle ABC, and then find the area of triangle OPC.
Step 3 Be able to represent a point on a straight line using letters (P-coordinates).
With the line OC as the base, find the y-coordinate of p, which is the height. P's coordinate is (t,-1/2t+4).
Step 4 Be able to represent a point on a straight line using letters (Q-coordinate).
Since the coordinates of P are (3,5/2), the line OP is y=⅚, and the coordinates of Q are (t,5/6).
Step 5 Be able to formulate an equation for area based on relationships among figures.
Finally, the area of △QAC was found from △AQO and △OQC, and the coordinates of Q were found.
We first defined the criteria for the system to infer the stuck points from the accumulated Handwriting Memo and self-explanatory sentences shown above. Using the handwritten data and self-explanatory texts of about 60 students, we tested whether they could be labeled by human hand and identify the stuck points. The number of operations characterized the handwritten data, and the self-explanatory texts were characterized by their similarity to the model answers in Table 3 and the description of features for analysis in Table 4. 60 samples were evaluated using five rubrics, and significant differences were found between the correct answers and the stumbling blocks as shown in Table 5, Figures   7 and 8. This study showed a significant difference between the correct answers and the stumbling blocks in each of the five rubrics.
Next, we proposed a deep learning-based model for automatically generating model answers to self-explanatory sentences. The model is based on a bottom-up approach and examines how the collected self-explanatory sentences can be used to generate model answer sentences with the necessary knowledge elements for the problem in question.
Sentence extraction was incorporated into the model. The results showed that 72% of the 25 questions used in the experiment were able to extract sentences after all knowledge elements (up to 5 elements) were satisfied (Nakamoto et al., 2022). Currently, the EXAIT system automatically generates model answers from the collected self-explanatory sentences, identifies the problem areas, and extracts the necessary knowledge elements. The similarity score estimated in Section 3.3, which corresponds to the 5-step rubric number.

Self-explanation length
The weighted average number of characters of two representative self-explanation sentences.
The weighted average of operation time associated with selfexplanation sentences.

Operation order
The weighted average of operation orders associated with a selfexplanation sentence.

Handwriting Frequency
The weighted average of frequency of ADD Handwriting associated with self-explanation sentences.

Add Memo Frequency
The weighted average of frequency of Typed Memo associated with self-explanation sentences.

Data-driven quiz recommendation and explanation for Mathematics
As a data-driven approach, the Bayesian knowledge tracing (BKT) model was employed (Corbett & Anderson, 1994), and implemented this model in the recommendation system and further developed a method to generate explanations of the reason why quizzes are recommended based on the parameters of this model. BKT models have been widely used to model student knowledge by calculating the probability that a student knows a skill at a given point in time based on their previous performance. The core parameters of the model are calculated during training or predefined heuristically based on expert knowledge, with the guess parameter representing the chance of giving a correct answer despite not knowing the skill, and slip of knowing a skill but giving a wrong answer. In our data-driven quiz recommendation, the guess and slip parameters of the BKT model for each question are calculated from the data of correct and incorrect answers for all questions for all students.
Quiz recommendations are made based on the probability that the student will correctly answer a question as determined by the BKT model where the student does not have an extremely high or low probability of correct answer.
A method of explaining the reason for recommending a particular quiz was developed using the guess and slip parameters of the BKT model so that the students understand why they should solve the recommended problems and proceed with their studies . In this explanation method, the recommended quizzes were categorized based on different feature types according to the values of the guess and slip model parameters and appropriate explanation texts are generated based on the interpretation of these feature types as shown in Figure 9. This explanation generation algorithm has been implemented into our recommendation system which is shown in the screenshot of the user interface in Figure 10. The explanation of the reason for the recommendation are displayed under the title of the recommended quiz. It was expected that students who see these explanations will be more incline to solve the quizzes due to the reason given, resulting in improved academic performance by actually solving the quiz. This explainable recommendation system was implemented and an experiment was conducted in high school mathematics classes comparing the use of explanations versus no explanations . It was found that there was a significantly higher click count for recommended quizzes that give explanations than those without explanations. In a post experiment survey to measure the student's perception of using the recommendation system, it was found that students in the group that were given explanations perceived that the recommendation was more convincing that those who did not receive explanations. The group that was given explanations also had higher perceived trust in the system, and there were fewer negative perceptions of the recommender system that in the group that did not receive explanations.
These results indicated the importance of the role of explanation for recommender systems in education.
In the perception survey it was found that some students were convinced by the explanations than others, suggesting that the explanations should be more tailored to the  individual student. Therefore, we also conducted research on student personalities to generate personalized explanations. To further this research, the examination of the extent to which individuals' personality can be predicted on the basis of 129 high school students' learning log data collected by the recommender system and Big Five personality trait survey data was carried out . A machine-learning approach was taken, and we were able to predict sub-groups of personalities to a curtain degree of accuracy, such as conscientiousness (R= 0.38), which is related to academic achievement. This result suggests the possibility to automatically segment people's personalities without the need for questionnaires, and then to provide optimal explanations and feedback for each segmentation to realize personalized educational recommendation explanation systems.

Model-driven quiz recommendation and explanation for Mathematics
To solve a math quiz, the student should have the necessary knowledge on the underlying concepts required in the math quiz (Birenbaum et al., 1993). Based on this idea, a modeldriven quiz recommender system with explanations was proposed in Dai et al. (2022).
Traditionally, recommender systems that utilize relation information on the knowledge concepts in educational materials and quizzes require hand coding by subject matter experts, which is a process that is costly and takes time . In this model, natural language processing methods were adopted to extract the knowledge concepts automatically from the contents of quizzes . Combined with the students quiz answers, a model of each students' mastery level on the concepts was estimated. Then, the recommendation and explanation was generated based on two criteria: 1) the quiz with appropriate difficulty, and 2) the quiz that may bring the largest potential learning gain. Figure 11 illustrates the process and an image of the Fig. 11 The overview of model-driven quiz recommendation and explanation for Mathematics recommendation and explanation. The recommender system was deployed in the LEAF system and has been tested in a Japanese high school class during a summer vacation. Some preliminary results and feedback were collected, and currently the model is being improved based on this feedback.

Knowledge map-based reading recommendation for extensive reading
Knowledge maps are graphs of knowledge concepts contained within a subject that it being studied, and shows the relation of how the components are related. Knowledge maps can be used for recommendation is based on a method proposed by  to automatically construct a vocabulary knowledge map generated from words included in learning materials. This method creates links based on the strength of semantic and contextual similarity between nodes of the map that represent vocabulary. Learning materials and books that contain the vocabulary from the map nodes are then linked, and the amount of engagement a student has had while studying a particular vocabulary can be estimated by analysis of learning behavior logs collected on the LEAF platform. A personal knowledge map that is weighted with the history of study engagement of vocabulary is created for each learner. As the map closely associates vocabulary that occurs in similar contexts, recommendations can be generated by identifying low weighted nodes that are adjacent to high weighted nodes which the learner has studied. An example of this method is shown in Figure 9, where logs that represent study behaviors, such as reading, listening, writing, and searching for the meaning of the vocabulary in a dictionary, are analyzed to estimate the amount of engagement with the word "until". If the node "until" has sufficient engagement when compared to other nodes in the knowledge map, learning materials and books that are associated with the adjacent nodes "became", "period", and "since" will be recommended for study to the learner as shown on the right of Figure 12.
The learner can then select from the list of recommended studies generated and shown in different parts of the system where students monitor and reflect on their studies, such as the learning analytics dashboard in LEAF called LogPalette. As the learner reads, listens, An English picture-book recommender system developed by  recommends picture-books that include as many words the learner should learn as possible.
By using the learning logs, it detects words each learner has learned and should be learned next. In other words, it finds words which the learner has read in picture-books in the vocabulary knowledge map and explores other books that include words that are adjacent to the words which have been read. The user interface of this recommender system shown in Figure 13 is implemented as a recommender feature of LogPalette. It shows at most 5 recommended picture-books and each book's recommendation weight, which means how highly it is recommended. When a learner opens the recommendation page implemented in LogPalette, the personal recommendation for the learner is displayed. When the learner selects the title of the books, they can jump to their BookRoll page and read them.
Another study by  suggested explainability of the recommendation to the reading recommendation in order to make it persuasive for learners. Although this proposed recommender system also recommends at most 5 e-books by using the same KMbased mechanism, the rationale of the recommendation is shown to the learners as shown in Figure 14. They proposed that it show some sentences that explain why the content was recommended to tackle trust issues, which may deprive the learners of motivation to learn.
The sentences inform the users that the recommended material includes words that they have encountered before in other e-books, quizzes, or a dictionary, which can be the rationale of the recommendations.

Discussion and conclusion
Recently, XAI has been gaining much attention in many different fields . Khosravi et al. (2022) focused on the Fairness, Accountability, Transparency, and Ethics of the use of xAI in education. While these are important topics in AI research in education and are a prime focus of xAI, we argue that there are also other important uses of the explanation of AI. In the context of education, we propose that there are some aspects of XAI that are unique to the field, such as the possibility of students learning from the explanations of a recommender. The implementation of AI system explanation and student self-explanation provides an opportunity to create a symbiotic learning system in which both the learner and AI system can benefit from explanation, which we proposed is called EXAIT. In the present paper, we present an overview of the proposed system, the basis for designing the EXAIT cycle as presented, and initial findings from research into the different stages of the system. The research into the EXAIT system offers a unique and original mutually beneficial cycle of recommendation explanation and answer selfexplanation based on AI which we see as an important step to creating learning systems that can co-evolve with learners. The current outputs of EXAIT have been on the individual stages of the cycle, and the fundamental infrastructure that has been designed to integrate into existing learning systems using standards such as xAPI and LTI. We consider that the overall design of the EXAIT cycle is an original contribution to the field, and in itself can be used as a template on which future symbiotic AI learning systems should be based. This approach to symbiotic learning systems differs from previous works in that the focus is on a mutually beneficial cycle as opposed to other systems that aim for AI system performance (Walsh et al., 2017;Wu et al., 2021), data collection (Zhang et al., 2022), or social aspects (Reidsma et al., 2016). In this section, we discuss the limitations of the current implementation and the challenges that should be addressed in future research. Firstly, in the current implementation, only individual components of the EXAIT cycle have been examined to assess their effectiveness. This has been due to the cycle consisting of several components which in themselves individually can be evaluated and refined as they are not directly based on pre-existing work that has undergone formal experiments to measure their effectiveness. This is an important step in the research and development of the EXAIT cycle, and we plan to conduct an experiment in future work to evaluate the sum of the effectiveness of the proposed system and its components. As such, individual components of the cycle also do not share information about the learners' characteristics with each other in the current implementation. It is anticipated that this will have the potential to further improve the effectiveness of the system, and therefore the effect of sharing learner profile information between the components should also be evaluated by formal experimentation. However, an empirical evaluation is necessary to determine the benefit of information sharing between different components of the EXAIT cycle, so that strengths and weaknesses can be identified and further improvements can be made as an ongoing iterative research development.
Secondly, while the practice of self-explanation by learners has been found to be beneficial in deepening understanding (Bisra et al., 2018), it is also a time-consuming process. We have also received feedback from teachers and students in regard to the workload, and it has been suggested that the self-explanation section of the EXAIT cycle should be regulated by the needs of the AI in understanding the student and also in taking up the students' time. Additionally, students that have not attained adequate knowledge of the domain or self-explanation skills might require additional regulation and scaffolding to ensure that they are not subject to the time-consuming self-explanation task as it could be counterproductive if they are ill-equipped even though there is much evidence given by Bisra et al. (2018) on the effectiveness of self-explanation. Therefore, two future areas of investigation exist: improving the support of the self-explanation system to shorten the time spent while ensuring it is still an effective practice for students, and identifying when it is beneficial for the system to have self-explanation from students to improve the system's understanding of the students current learning state while taking into consideration the students current state.

Future possibilities of EXAIT
This article presented the underpinnings and development of the EXAIT symbiotic learning system which is characterized by the use of both student self-explanation and recommendation explanation. The purpose of the system and the learning cycle around which is has been designed is to enhance both the systems understanding of the students current learning state, and also increase the student's awareness of their learning through system explanation and practicing self-explanation of their answering process. Currently, the system has been implemented in Japanese secondary schools for Mathematics and English subjects, however it is not limited to those institutions or subjects and in the future broader application should be investigated. There is potential to explore the application of self-explanation in subjects other than mathematics which has historically been the main focus of previous research. While the personalization of recommendations and their explanations for personality traits has been investigated, there is still potential to investigate other facets of student characteristics for greater personalization of recommendations and explanations. The generalization of models presented in the present paper should also be investigated, so that the EXAIT system can be easily applied to other subjects or different levels of education, such as: elementary, higher, open and corporate education. In future work, we plan to formally evaluate learner-centric and content-centric methods of recommendation explanation. We also plan to implement and evaluate the effectiveness of computer-generated scaffolding for self-explanations by students. While EXAIT currently engages the learner in self-explanation for the benefit of themselves and also the system, there is also potential to expand this to a knowledge-sharing mechanism and serve as an intellectual commons between learners (Eikeland, 2013).
Authors' contributions HO, BF, KT, YD, RN and KT contributed to the research conceptualization and methodology. BF, KT, YD, RN and KT wrote the manuscript. BF, KT, YD, and HO provided comments to improve the manuscript. All authors read and approved the final manuscript. Kyosuke Takami is a senior researcher at Education Data Science Center, National Institute for Educational Policy Research (NIER) in Japan. He has prior expertise in the application of neuroscience in education and received a PhD degree from Osaka University. He has been a high school teacher for more than 9 years. His research interests include: Learning Analytics, Educational Data Mining and Educational Technologies.

Authors' information
Yiling Dai is a Program-Specific Researcher at the Academic Center for Computing and Media Studies, Kyoto University. She received a Bachelor's degree from Zhejiang University, a Master's degree from the Graduate School of Business, Rikkyo University, and a PhD degree from the Graduate School of Informatics, Kyoto University. Her research interests include: Information Retrieval, Knowledge Discovery, Educational Data Mining and Learning Analytics.
Ryosuke Nakamoto is currently a doctoral student at the Graduate School of Informatics, Kyoto University. His research focuses on using the collected self-explanations and examining how they can be used to support learning, including identifying students' stuck points and feedback to students in mathematics.
Kensuke Takii is currently a doctoral student in the Graduate School of Informatics, Kyoto University. His research interest includes learning analytics and learner modeling, aiming at building a learning analytics framework for modeling learners' knowledge states and realizing personalized learning environments.