Evaluating the kit-build concept mapping process using sub-map scoring

Concept mapping allows learners to visually represent their knowledge by connecting nodes (concepts) and links (relations between concepts). The kit-build (KB) concept map framework enhances this process by enabling learners to recompose a concept map from provided nodes and links, leading to improved learning outcomes. Additionally, KB employs an automatic assessment method called “ Full Map Scoring (FMS) ” , which evaluates the learner ’ s understanding based on the recomposed concept map. However, FMS only evaluates the final product of the recomposition activity, neglecting the process itself. This is a potential limitation because different processes leading to the same result could reflect different levels of understanding among learners. Therefore, it is crucial to incorporate process analysis into learner assessment. To address this issue, our research proposes a new assessment procedure termed “ Sub-Map Scoring (SMS) ” . A concept map is generally composed of several sub-maps with each sub-map representing a set of meanings. We hypothesize that if a learner comprehends the meaning of a sub-map, the learner will recompose the sub-map as a continuous activity. Therefore, SMS evaluates the recomposition process of each sub-map from the viewpoint of continuity, and the overall SMS score is derived from these evaluations. To verify the effectiveness of SMS, we compared SMS and FMS scores using data from a practical use of the KB framework. A multiple linear regression analysis confirmed that the SMS score was a more precise predictor of learning gain than the FMS score.


Introduction
A concept map is a well-known visual thinking technique popularly used in education (Jonassen et al., 2013;Novak, 1990;Novak et al., 1984).In an educational context, it is popularly used to request learners to visually represent their understanding of an object of learning (Cañas et al., 2016;Dwi Prasetya et al., 2020).Many investigations have reported that the activity of representing understanding with the concept map enhances learning more effectively than traditional learning methods (Elhelou, 1997;Horton et al., 1993;Lee et al., 2013).
Moreover, various studies have been conducted using the concept map as an assessment tool to evaluate learner's understanding (Gregoriades et al., 2009;Kinchin et al., 2000;Vanides et al., 2005).Most of these studies have relied on manual assessment methods conducted by human raters and have reported their usefulness.However, manual assessment methods present two issues, they are time-consuming and lack the stability of results among raters (Papajohn, 2002;Ruiz-Primo & Shavelson, 1996).As a solution of these problems, automatic assessment has been investigated as a promising approach (Wunnasri et al., 2018b).
Kit-build concept map (KB) is a framework that realizes automatic assessment of concept maps made by learners (Hirashima et al., 2015;Yamasaki et al., 2010).In KB, learners are provided with a set of components generated by decomposing a concept map created by a teacher, which is made as an ideal representation of the understanding learners should achieve.The learners are then requested to recompose a concept map by connecting the provided components.The map recomposed by the learner is automatically assessed by comparing it with the original teacher's map.
The validity of the automatic assessment by KB has been confirmed through comparisons with manual evaluation methods (Wunnasri et al., 2018b).Additionally, the usefulness of the diagnosis for additional teaching by teachers (Pailai et al., 2017;Yoshida et al., 2013), automatic feedback by learning support systems (Furtado et al., 2019;Pailai et al., 2018), and pair discussion (Sadita et al., 2020;Wunnasri et al., 2018a) have been investigated through experiments and practice.
However, the assessment of KB is specifically conducted only for the result of the recomposition, and the process of the recomposition itself is not the target of the assessment.
Therefore, if learners recompose the same map, but with different recomposition processes, they will receive the same assessment result.Several previous investigations, however, have highlighted that in the context of a learning activity, similar outcome can be led to a different level of understanding when the process is different (Ford et al., 1998;Lee & Fortune, 2013;Van Rossum & Schenk, 1984).As noted by Marton and Säljö (1976), process analysis could reveal relations to the degree of understanding and learning gain.
Especially, because of the nature of KB that enables learners to recompose the same map in various ways, the process assessment is an important research issue to adequately evaluate learner's understanding.
In this study, we focus on the idea that a concept map can be interpreted as a combination of several sub-maps, each representing a set of meanings (Kinchin & Alias, 2005;Roberts, 1999).Based on this idea, it is reasonable to assume that learners will recompose the sub-map through a series of activities if they have a good understanding of the set of meanings.
Based on this assumption, it becomes possible to evaluate the process of sub-map recomposition.If this assumption and approximate evaluation of sub-map are adequate, it becomes possible to assess the process of the whole map recomposition as the sum of the assessments of each sub-map recomposition process.The assessment method used in previous research, targeting the whole concept map as an outcome of recomposition, is called Full Map Scoring (FMS for short).The new method proposed in this paper, focusing on recomposition processes of sub-maps is called Sub-Map Scoring (SMS).
SMS has been implemented and evaluated in comparison with FMS.The evaluation was conducted using data obtained from the practical use of KB in a university class with 60 second-year undergraduate students.The learning gain as the normalized differences between pre-test scores and post-test scores was used in this evaluation.Hence the first research question (RQ1) is: Can SMS assess the process of KB concept mapping activity in terms of its relationship with the learning gain?The second research question (RQ2) is: Is SMS a better predictor of learner's learning gain compared to FMS?This paper's Background subsections explain the following: (a) the basics of a concept map as an assessment tool, (b) the KB concept map as the framework for concept map recomposition and automatic diagnosis and (c) sub-map in a concept map.The Sub-Map Scoring section discusses the approach used in assessing the KB concept map recomposition process.The Experimental Evaluation section describes the participants, instruments, procedures, and analysis of the experiment.The Results and Discussion section centers on the experiment results and deliberation of our findings.Lastly, the Conclusions and Future Works section summarizes this research and discusses future works.

Concept map as an assessment tool
Concept maps are represented by graphs and consist of two symbols: nodes represent concepts and links represent the relationship between concepts (Cañas et al., 2016;Dwi Prasetya et al., 2020).In the context of learning subjects and reading materials, relationship between concepts represents proposition.In educational context, concept map is a widely used technique for learning and assessment (Hirashima et al., 2015;Plotnick, 1997;Vanides et al., 2005;Zheng et al., 2019).Several investigations have mentioned the advantages of concept map, including the ability to enable knowledge sharing (Novak & Symington, 1982), improve learner's creativity and motivation (Chan, 2017), improve critical thinking skills (Sundararajan et al., 2018), and actively engage learners in the learning process (Novak et al., 1984).
As an assessment tool, many methods have been proposed to utilize concept maps to assess learner's knowledge.Several research have confirmed the validity and reliability of using concept map as an assessment tool (Plotnick, 1997;Vanides et al., 2005;Zheng et al., 2019).There are two main approaches in assessing the concept map: one is structural scoring and the other is relational scoring.A structural scoring was proposed by Novak et al. (1984).This method works by assessing the correctness of hierarchy level and crosslink validity which reflects learner's ability in creative thinking (West et al., 2000).Relational scoring, on the other hand, considers the meaning of propositions instead of the map structure (McClure & Bell, 1990).This method works by evaluating the possibility of relationship between each proposition, appropriateness of concept labels and compatibility between labels.McClure et al. (1999) conducted a study between the two methods by requesting 63 students to construct a concept map using 20 provided concepts and creating the links to connect the concepts.A group consisting of 12 raters then scored each map separately.
Result shows that the relational and structural scoring method has a close relation when a criteria map, or teacher-build map is used as a scoring reference.Kit-build concept map also uses the expert map as a criteria map for assessment (Hirashima et al., 2015).The automatic assessment in the KB map is explained in the following subsection in more detail.

Kit-build concept map
The kit-build concept map is a concept-mapping framework that enables the automatic diagnosis of a learner's concept map (Hirashima et al., 2015) that consists of teacher's map building, learner map building, and KB analyzer.In this framework, there is an activity in which the learner is tasked with recomposing the concept map components (kit), which consists of nodes and links, into a complete map.The kit was prepared by decomposing the teacher's map, which is kind of expert map created by a teacher for the learning material.
In KB concept map, learners recompose the teacher's map instead of creating their own concept maps from scratch.
In the kit-build concept map, the goal-map building is divided into two tasks: segmentation and structuring (Yamasaki et al., 2010).The segmentation task involves extracting the nodes and links (kit) of a concept map from the learning materials, while the structuring task involves reconnecting the extracted elements (kit) and creating a complete concept map. Figure 1 illustrates how the kit (i.e., concepts and links) was extracted from the learning materials or resources during the segmentation task and how the kit's elements were connected to form the concept map during the structuring task.The next step following the segmentation and structuring task, the goal map is decomposed into a kit of separated nodes and links, which is then given to a learner.The learner recomposes the nodes and links into a map which is called "learner map" (structuring task).After the learners complete the map recomposition, the system automatically compares the learner's map to the goal map in terms of their similarities.The diagnosis works by superimposing the learner map over the goal map to detect which and how many links are excessive (i.e., incorrectly connected), missing (i.e., should be connected but not), and matching (i.e., correctly connected) (Hirashima et al., 2015).
Figure 2 shows the kit-build concept map system's lifecycle flow.In the Phase 3 of the The framework also allows the automatic diagnosis of several learner maps, known as group maps (Hirashima et al., 2015).In this diagnosis, multiple learner maps are combined into a group map, which is then overlaid with the goal map.By leveraging this kind of diagnosis, teachers can determine which parts of the learning material are not properly understood, misunderstood, and accurately understood by the majority of the learners and receive apt feedback to assess the learning condition.
KB has advantages over scratch-build (SB) concept map as suggested by several research (e.g., Alkhateeb et al., 2015;Andoko et al., 2020;Funaoi et al., 2011).Compared to SB, components to build the concept map are provided in the KB system and it allows analysis on the recorded learner's activity, including the connected propositions in relation to the teacher's map.Therefore, the proposed method in this study was applied to the KB concept mapping.

Sub-map in a concept map
In a concept map, propositions that share similar meanings and belong to a specific domain of knowledge can be grouped together into a sub-map.A sub-map typically comprises propositions that have related meanings are derived from the same subdomain of knowledge.According to Gerstner and Bogner (2009), sub-maps are defined as groups of propositions originating from distinct domains within the learning material.Schneider et al. (2021) states that dividing a concept map into a series of sub-maps can be accomplished by focusing on a specific knowledge domain.This approach aligns with the spatial continuity principle described by Moreno and Mayer (1999) and Schroeder and Cenkci (2018), which suggests arranging propositions that share similar information in close proximity to one another within the concept map.
Additionally, segmenting the learning material into meaningful and coherent segments, as recommended in the segmenting principle, can enhance learning outcomes (Schneider et al., 2021).In a study conducted by Kinchin and Alias (2005), a sub-map is described as a collection of propositions with meaningfully related concepts.The study highlights that a teacher created the sub-map to provide a concise overview of the topic's content.
Defining the sub-map can also aid in identifying prerequisite knowledge necessary to understand a particular sub-map effectively.
Fig. 2 The kit-build concept map system's lifecycle

Sub-Map Scoring (SMS)
The kit-build concept map has already incorporated the Full Map Scoring (FMS) to score the concept maps recomposed by learners.The map recomposed by a learner is called a learner map.This scoring method compares propositions in a learner map to those in the goal map.Because both maps are composed of the same components, if a proposition exists in both maps, the proposition is judged as a correct proposition.If a proposition exists only in the learner map, it is judged as an excess proposition.A proposition existing only in the goal map is judged as a lacking proposition.The score would be calculated by counting the number of correct propositions and comparing it to the number of propositions from the teacher's map.For example, when the number of correct propositions is five and the number of the teacher's map's propositions is ten, the FMS score is 50.
In contrast to FMS, SMS has been proposed based on the idea that the structure of a concept map consists of several sub-maps, a sub-map is a subset of propositions with tightly related meanings.The proposed method investigates the learner's learning gain based on recomposition process of each sub-map.As a preparation of SMS, a teacher is required to define several sub-maps in the teacher's map.For each sub-map, SMS evaluates the learner's recomposition sequence of propositions belonging to the sub-map.Figure 3 illustrates the sub-map specified by the teacher in the goal map, while Figures 4 and 5 depict the sequence of propositions recomposed by learners.Figure 3 shows five sub-maps (SMs) defined by a teacher.Sub-map 1 (SM1) is made up of two propositions: proposition 1 (P1) and proposition 2 (P2), which are written in the parentheses at the end of a link label.SM2 comprises P3, P4, and P5, while SM3 involves P12 and P14.SM4 has P7, P8, P9, and P10, while SM5 covers P13 and P15.This sub-map configuration can be written as follows: {P1, P2}{P3, P4, P5}{P12, P14}{P7, P8, P9, P10}{P13, P15}.In contrast, Learner B completed the sub-maps in discontinuity, as shown in Figure 5 (bottom).In the figure, Learner B connected P1, P3, P2, P4, P12, P14, P5, in which only P12 and P14 were connected consecutively that belong to a sub-map (SM3).These processes imply that Learner B comprehends the set of propositions as a unit of knowledge less than Learner A. Therefore, Learner A's process score should be higher than that of Learner B.
In the context of formative assessment, teachers could use the process assessment to evaluate learner's process in the activity of recomposing the concept map.For example, the teacher may encourage Learner B to further enhance their understanding of the learning material.The discrepancy in connected propositions suggests that Learner B might not be effectively elaborating on the knowledge implied in a set of propositions that share similar meanings.
Based on the above consideration, we proposed Sub-Map Scoring (SMS) as the recomposition process's scoring method.To calculate the SMS, we analyzed the propositions' sequence connected by the learner.In each sub-map, we scanned the learner's proposition that is sequentially connected and matched the sub-map's propositions.Figure 5 illustrates an example of the analysis.In the teacher's map, there are three According to this data, the matched proposition sequence with the sub-maps is {P1, P2} for SM1, {P3, P5, P4} for SM2, and {P14, P12} for SM3.The method treats the propositions in a sub-map as a set, therefore, the percentage of matched propositions sequence for each sub-map is then calculated.If a sub-map is composed of four propositions, two of which are recomposed in continuity, the sub-map's score is calculated as (2 / 4) * 100 = 50.The proposed method assumes that all the sub-maps have equal weight of importance, so the total SMS is the average of each sub-map score.This percentage is the rate of the number of propositions connected as a set, in comparison with the number of propositions in a sub-map.This is to represent the rate of sub-map completeness.The complete calculation for the two learners is as follows: In the case of a sequence with recurring propositions set that belongs to a sub-map, the selection is based on the set with the greatest number of propositions.This is because a larger set of connected propositions is assumed to better reflect an understanding of the knowledge context conveyed in the sub-map.For example, a sub-map 1 (SM1) consists of {P1, P2, P3, P4, P5} and a learner made a sequence of {P1, P2, P9, P10, P3, P4, P5}.In this sequence, {P1, P2} matched the SM1, and the next sequence is {P9, P10}, is no longer matched SM1.However, the next sequence is {P3, P4, P5}, which recurringly matched the SM1.In this case, the sequence of {P3, P4, P5} is counted to be scored.

Experimental evaluation Participants
The subjects of this research are 60 second-year undergraduate learners from the Information Technology Department of the State Polytechnic of Malang, Indonesia.The students have no prior knowledge or experience building a concept map.Those who did not complete the experiment activities, did not attend the experiment sessions due to a system error, or left the question tests blank were not included.After the teacher delivered the learning material presentation, the learners used the kit-build system.

Instruments
The experiment's instruments included the learning module used in the Object-Oriented Programming (OOP) course.This module was compiled and released by the OOP course teaching team (four teachers) from the Information Technology Department of the State Polytechnic of Malang, Indonesia.One of the team members is the first author.The module material complied with the standard OOP education provided by the ORACLE academy program.The OOP concepts used were extracted from the module's encapsulation topic.
The kit-build goal map used for this experiment consists of 15 propositions with 14 concepts, 15 links, and five sub-maps, as seen in Figure 3.The goal map was proposed by this paper's first author and presented to the three OOP teaching team members.The goal map was then discussed and modified according to the teaching team's inputs.The pretest, posttest, and delayed test employed the same 20 multiple-choice questions, all of which were collaboratively prepared by the teaching team.

Procedures
The experiments took a total of 60 minutes.The learners were first given an introduction and tutorial on how to use the KB system.At first, learners were given a pretest to measure their learning knowledge baseline.The pretest was completed in 10 minutes.Afterward, the learning material was delivered in the form of a 15-minute teacher presentation.Then, the kit-build concept mapping activity using the KB system followed.In the KB system, learners were tasked to reconstruct the kit or concept map components (concepts and links) into a learner map, which took 15 minutes to complete.A 10-minute posttest then followed.As mentioned, the questions on this test are the same as those on the pre-test.A week later, using the same questions, the delayed test was administered in 10 minutes.During the pretest, posttest, and delayed test, all the materials were closed.

Analysis
In this research, six variables were used to perform the analysis: pre-test score, post-test score, delayed-test score, learning gain, SMS score, and FMS score.The learning gain was calculated and used to evaluate the learner's understanding by measuring it based on pretest score and post-test score using normalized gain formula, introduced by Hake (1998).
The normalized gain measurement has some significant advantages over the regular gain calculation in which gain = post-test scorepre-test score (Bao, 2006;Coletta & Steinert, 2020;Marx & Cummings, 2007).Particularly, it allows teachers to compare increases in scores for many learner populations by taking the pre-test score into consideration and thereby focusing on the possible improvement which makes learners with different pre-test scores more comparable.The normalized gain formula is as follows: The learning effect of the KB concept mapping activity was evaluated by analyzing significance of differences between pre-post-delayed test score using analysis of variance (ANOVA).Shapiro-Wilk test was performed on the test results and resulted in a normal distribution on the pre-test scores (p-value = 0.2705), non-normally distributed data on the post-test scores (p-value = 6.774e-06) and delayed-test scores (p-value = 0.006).Therefore, a non-parametric ANOVA using Kruskal-Wallis test was performed to evaluate the KB activity effect.A p-value of < 0.05 considered statistically significant differences between scores.
The correlation test was performed to evaluate the relationship between SMS, FMS, posttest score, delayed-test score and learning gain using Pearson's r coefficient.To interpret the r-value, we use Schober et al. ( 2018) criteria of r < 0.39 being weak, r > 0.40 and r < 0.69 being moderate, and r > 0.70 being strong correlation.Multiple linear regression analysis was performed to evaluate between SMS and FMS in terms of predicting learner's learning gain.

Kit-build activity effect analysis
Before evaluating the SMS, we performed the KB activity effect analysis to confirm that the activity produced adequate outcome that was observed in previous research (e.g., Andoko et al., 2020;Dwi Prasetya et al., 2020;Funaoi et al., 2011;Hirashima et al., 2015;Pailai et al., 2017;Yamasaki et al., 2010).This analysis compared the pre-post test scores and the pre-delayed test scores to investigate the KB concept mapping's activity effect.
Figure 6 shows the test score visualization of the pre-test, post-test, and delayed-test scores.
The mean analysis results are depicted in Table 1.
The mean analysis results in Table 1 show that the pre-test mean score was 33.960, the post-test mean score was 78.118, and the delayed-test mean score was 73.633.To assess the significance of the pre-test and post-test mean scores, as well as the post-test and delayed-test mean scores, a nonparametric ANOVA using Kruskal-Wallis analysis was conducted, and the results are displayed in Table 2.With a p-value < 0.001, the post-test score is statistically significant than the pre-test score.Post-test and delayed-test scores were not statistically significant, with a p-value = 0.194, p > 0.05, as shown in Table 2.

Correlation analysis
To answer the first research question (RQ1), correlation test of learning gain with the SMS and FMS was used to evaluate the relationship between the scores with learning gain.The results are shown in Table 3.The correlation value between the SMS and learning gain was 0.711, indicating a strong positive relationship, while the value between the FMS and learning gain was 0.674, indicating a moderately positive link.Both are statistically significant with a p-value < 0.05.

Multiple linear regression analysis
To answer the second research question (RQ2) in discerning how the scores (i.e., SMS and FMS) predict the learner's learning gain in terms of the learning material delivered by the teacher, a multiple linear regression analysis (MLRA) was performed.
First, the pre-test score was examined to determine whether it also predicts learning gain.
Regression analysis was performed on the pre-test score and FMS, as well as on the pretest score and SMS.The first test of pre-test score and FMS to predict learning gain, results showed that pre-test has a p-value of 0.392, FMS has a p-value of less than 0.05 with R 2 = 0.461.In the second test of pre-test score and SMS to predict learning gain, results showed that pre-test has a p-value of 0.173, SMS has a p-value of less than 0.05 with R 2 = 0.521.This indicates that when compared to FMS and SMS, the pre-test score cannot be used to predict the learning gain.
Second, the pre-test score, SMS, and FMS were compared to discern which among the three variables is the better predictor on the learning gain value.Table 4 shows that SMS, with a p-value of 0.00959, p < 0.05, and R 2 = 0.5226718, has a significant association with learning gain when compared to the pre-test score and FMS.
The Durbin-Watson test was performed to check the assumption that the residuals from the MLRA model are independent.The test resulted in d¬ -value of 1.842 indicates that there is no autocorrelation problem.The p-value of 0.386 indicates that the residuals are independent and therefore the predictor variables are statistically significant.A further test on homoscedasticity using Non-constant Variance Score Test was performed with resulted in p-value of 0.148.This indicates that the residuals are homoscedastic, therefore the residuals have equal variance for every value of the fitted values and of the predictors.

Supplementary analysis
Additional analysis to evaluate the relationship between SMS, post-test score, delayed-test score and FMS was performed to confirm the usability of SMS in terms of assessing the learning activity outcomes and map quality.
To evaluate the relationship between learner's understanding in terms of the post-test score and SMS, and the FMS and post-test score, a correlation test was performed.The results showed that the SMS and FMS have a similar correlation value, as seen in

Research question 1 & 2
As shown in the Kit-Build Activity Effect Analysis subsection, the results from the learning activity effect analysis confirmed that the KB activity outcomes were in-line with previous research, with a significant increase between pre-test and post-test scores.Therefore, the activity resulted in a meaningful outcome, hence it is adequate to proceed in analyzing the SMS based on these outcomes.
SMS, as well as the FMS, have a high correlation with learning gain with correlation value of 0.711 and 0.674 respectively.The FMS result is in-line with previous research that confirmed the relation between KB map score method with learner's understanding (Hirashima et al., 2015;Wunnasri et al., 2018b;Yamasaki et al., 2010).Based on this, it is implied that SMS and FMS can both be used as scoring systems to assess learner's learning gain.
Furthermore, multiple linear regression analysis shows a p-value of 0.00959 for SMS, when tested against FMS and pre-test score.This means that SMS is a better predictor of learning gain than FMS and pre-test score.Because learning gain is closely connected to the process of KB concept mapping activity, this suggests that the SMS is effective in evaluating the process in KB concept mapping.
A correlation value of 0.719 between post-test and SMS and 0.707 between post-test and FMS means that both SMS and FMS have a similar relationship in reflecting learner understanding represented by the post-test score.In terms of memory retention, a correlation value of 0.588 between delayed-test and SMS and 0.550 between delayed-test and FMS indicates that the SMS and FMS have the same relationship in reflecting the learners' memory retention.The similar relationship of SMS and FMS towards post and delayed test score indicates that learner's process in KB concept mapping affects the understanding and memory retention as well as the learner's map quality.Furthermore, the result of correlation analysis between FMS and SMS confirmed that learner's process in concept map building has a relationship with map quality (Chiu & Lin, 2012;Srivastava et al., 2021).Based on these results, it is implied that a good process in KB concept mapping could lead to not only a better learning gain, but also a better understanding and memory retention of the learning subject, and better map quality.Therefore, supporting the process in KB concept map recomposition could produce an adequate outcome in the activity.

Formative assessment based on process assessment
In the context of formative assessment, teachers could use the SMS as an assessment to evaluate the learner's knowledge elaboration.Knowledge elaboration has been defined as the interconnection, restructuring and integration of new information with prior knowledge (Weinstein & Mayer, 1986;Zheng et al., 2022).In comparison with learner's understanding which reflects comprehension and interpretation of an information, knowledge elaboration refers to the process of building on existing knowledge to deepen one's understanding, make connections between different pieces of information, and apply knowledge in new and more complex situations.Good process of knowledge elaboration allows coherent structure of knowledge organization and are essential for meaningful learning (Kalyuga, 2009).
In a concept mapping activity, a process of knowledge elaboration can be evaluated in the process of concept map recomposing activity.This can be achieved by assessing how learners connect propositions as a process of organizing, restructuring and interconnecting concepts using their knowledge.In a study about knowledge acquisition and recall, Bergman et al. (2015) mentioned that the learner's ability to create a meaningful connection between propositions in a relevant context shows a good elaboration of learner's knowledge.In SMS, propositions that have a relevant context are grouped into a sub-map, and learners are scored based on the continuity of the connected propositions in a sub-map.
Therefore, high SMS means that the learners are able to create a meaningful connection between propositions in a relevant context, hence have a good process of knowledge elaboration.
Moreover, even though the scoring method of SMS is different with FMS, the SMS could result in a similar correlation between FMS and learning gain.However, there are several cases, for example of learners with high FMS but low SMS.Or in an extreme case, a full FMS but zero SMS.This indicates that the learner can recompose a good concept map but may lack knowledge elaboration as reflected by their recomposing process.As SMS is the better predictor for learning gain than FMS, means that the process of knowledge elaboration plays an important aspect in the outcomes of learning gain.High correlation between SMS and FMS also depicts the relationship between knowledge elaboration process and map quality.This is one aspect that is potentially realized by SMS, in comparison with assessing the final concept map score realized by FMS.Therefore, it is a good approach for teachers to monitor learners and guide them to a better process in the concept map recomposition activity.
Thus far, there is no support for the recomposition process in KB concept mapping.
However, if the sub-map recomposition behavior affects learning gain, then it is necessary to design a recomposition in-process support based on the sub-map structure.Consequently, some of the important future tasks include designing such support and evaluating its effectiveness.A more thorough investigation on the evaluation of knowledge elaboration and its relations with the concept mapping activity could also reveal more aspects of learner's understanding.

Figure 2
Figure 2 illustrates the diagnosis.Blue-solid line means excessive links, red-dashed line means missing links, and green line means matching links.

Fig. 3
Fig. 3 Sub-map specified within a teacher's map

Figures 4
Figures 4 and 5 illustrate two examples of different proposition recomposition sequences.The order of proposition recomposition is represented by numbers in rectangles, such as O1 or O2.O1 means the first connected proposition, O2 means the second connected proposition, and so on.For the sake of clarity, only the propositions used from O1 to O7 are shown.Figure 4 displays the work of two learners (Learners A and B) who recomposed the same final map, with their recomposition processes differing in the proposition sequence order from O1 to O7.Since the two recomposed maps in Figure 4 are identical, both recomposed maps get the same FMS score although their recomposition processes are different.SMS is able to give difference scores for the two recompositions.By using these examples, the procedure is explained concretely.It is assumed that Learner A completed each of the three sub-maps in continuity, as shown in Figure 5 (top).In the figure, Learner A connected P1 and P2, which belong to sub-map 1 (SM1).Consecutively, P3, P5, P4, which belong to SM2, then P14, P12 which belong to SM3.The manner in which Learner A connected propositions that belong to a sub-map consecutively indicates a good understanding of the set of propositions as a unit of knowledge.When a learner organizes and presents the propositions in a coherent and consecutive manner, it suggests that they grasp the relationships and connections between those propositions.

Fig. 4
Fig. 4 Sequence example for Learner A (left) and Learner B (right)

Fig. 5
Fig. 5 Sequence path summary example for Process A and Process B

Fig. 6
Fig. 6 Pre-test, post-test, and delayed-test scores Table 1 summarizes the experimental data, showing the mean and standard deviation values of the pre-test score, post-test score, delayed-test score, learning gain, SMS, and FMS.

Table 1
Mean and standard deviation of the pre-test score, post-test score, delayed-test score, learning gain value, SMS, and FMS

Table 2
Kruskal-Wallis analysis results

Table 3
Correlation between the SMS and learning gain, and the FMS and learning gain

Table 4
Multiple linear regression analysis on the pre-test score, SMS, and FMS in relation to learning gain

Table 5 ,
with r coefficients of 0.719 and 0.707, respectively, both of which indicate a strong positive correlation and are statistically significant with a p-value < 0.05.The delayed-test score was used to measure the learner's memory retention after one week of receiving the learning material.The correlation between the SMS and delayed-test score, and the FMS and delayed-test score were also assessed to evaluate how SMS and FMS relate to memory retention.The results are depicted in Table6.The correlation value of the SMS and delayed-test score was 0.588, which was moderately positive; similarly, the correlation value of the FMS and delayed-test score was 0.550, which was also moderately positive.Both have a statistically significant correlation with a p-value < 0.05.A correlation test between SMS and FMS was conducted to confirm the usability of SMS in relation to FMS and to evaluate that learners who have a good process on the KB concept mapping activity means they will result in a good concept map quality.Table7depicts the relationship between SMS and FMS.The two variables are highly correlated with a

Table
Correlation between the SMS and post-test score, and the FMS and post-test score

Table 6
Correlation between the SMS and delayed-test score, and the FMS and delayed-test score

Table 7
Correlation between the SMS and FMS