Effects of Different Types of Cues and Self-Explanation Prompts in Instructional Videos on Deep Learning: Evidence from Multiple Data Analysis

The purpose of this study was to investigate the effects of different types of cues and self-explanation prompts in instructional videos on intrinsic motivation, learning engagement, learning outcomes, and cognitive load, which were indicators to measure deep learning performance. Seventy-two college students were randomly assigned to one of the six conditions in a 3 × 2 factorial design with cues (visual vs. textual vs. combined textual-&-visual) and self-explanation prompts (prediction vs. reflection) as the between-subjects factors. To measure participants' learning engagement, Neurosky mindwave mobile and Tobii pro X3-120 eye-tracker were used to collect their brain wave data and eye movement data, respectively. Learning outcomes were measured with retention and transfer tests, and questionnaires were used to measure participants' intrinsic motivation and cognitive load, respectively. The results revealed that the textual cues significantly facilitated learning outcomes and learning engagement—attention–while the reflection prompts significantly affected learning engagement—the mean fixation duration—and cognitive load. Notably, the combination of textual cues and reflection prompts and the combination of visual cues and prediction prompts allowed the participants to focus and engage in the video learning process more deeply, resulting in a significantly higher learning outcome than their peers from other conditions. This research could provide some implications for designing short instructional videos to facilitate deep learning.

Keywords: Cues; Self-explanation prompts; Deep learning; Cognitive processing; Instructional video

Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Nowadays, instructional video has become one of the most common and powerful digital learning resources. Especially the instructional video with the feature of short duration to present information visually and vividly, which learners of all ages favor (Wang et al., [67]; Zhu et al., [76]). Notably, the high-quality instructional videos could guide learners' attention to specific information in the complex dynamic materials being shown, helping them build a complete psychological model and accurately understand the objects, knowledge, and causality in visual representation (De Koning et al., [22]; Wang et al., [67]). That is, well-designed instructional videos can bring deeper learning performance, which is learners could produce a positive and internal incentive experience, so naturally, they with a deep focus will be more engaged and facilitate their knowledge retention and transfer ability (Aguiar-Castillo et al., [1]; Baeten et al., [7]; Grover et al., [28]). However, when students use instructional videos with challenging content or unreasonable presentation of the complex content if without providing them the necessary learning scaffold, they may have a high cognitive load, mind straying, and surface cognitive activities. For instance, Bayraktar et al. (2019) found that the instructional videos without necessary cues related to the critical information easily lead to students ignoring them instead of focusing on the entire materials being shown, resulting in change blindness, which is not beneficial to deeper learning performance. According to Instructional Transaction Theory, high-quality instructional videos should guide students to pay attention to critical information and foster them to engage actively in deep cognition activities and construct meaning (Merrill, [48]). Therefore, embedding some necessary prompting signals for the vital information in instructional videos development or providing learning scaffolds when students learn with the videos may help promote their deep learning performance.

Previous studies have demonstrated that the cues and self-explanation prompts might potentially facilitate students' learning in the multimedia environment (Lin & Atkinson, [40]; Lin et al., [41]). In addition, Wang et al., ([67]) explored the effects of cues in the short instructional video with eye-tracking technologies. They found that the added cues had improved students' retention knowledge scores. Building on these previous researches in the current study, we investigate the effects of different cues (visual vs. textual vs. combined textual-&-visual) and self-explanation prompts (prediction vs. reflection) on students' deep learning in instructional videos. Moreover, we adopt a multiple data method to evaluate the impact of these two strategies, cues and self-explanation prompts, on students' deep learning performance, such as intrinsic motivation, learning engagement (e.g., cognitive, emotional, and behavioral engagement), learning outcomes, and cognitive load.

In recent decades, educators and researchers have paid much attention to promoting the effective occurrence of students' deep learning. To clearly understand the deep learning approach, they usually compared it with the surface learning approach that aims to rot memorization or simple recollection of learning content. The occurrence of deep learning requires students to understand and construct meaning critically, seek principles to organize learning information, find relationships among learning materials, and apply knowledge to new situations smoothly (Akyol & Garrison, [2]; Filius et al., [25]; Koszalka et al., [37]). However, students learning with a surface approach generally remember existing information, absorb new information at a primary level, and understand knowledge at a superficial level, but rarely involve high-level thinking activities (Offir et al., [54]). Notably, previous studies have reported that students were likely to produce high-quality learning outcomes with the deep learning approach in digital learning settings. For instance, Chen et al., ([17]) used the collaborative concept mapping as an effective learning scaffold strategy to foster the occurrence of deeper learning. They found that students had a more profound exchange of ideas, better knowledge convergence, and higher problem-solving skills as a result of participating in deep learning with collaborative concept mapping strategy. Furthermore, Wang et al., ([66]) established that deep learning triggered by problem-solving could enable students to manage complexity and challenges effectively and further promote their high-level learning participation and performance.

Deep learning performance is involved in both internal cognitive processing and external learning outcomes for students. Wang et al., ([66]) stated that deep learning is characterized by a high level of engagement in learning, driven by intrinsic motivation sustained engagement and achieving a high level of understanding and performance. Previous studies have found that students who use a deep learning approach may have more internal motivation, driving them to seek and obtain meaning with a high level of engagement and achieve a deeper understanding and good performance (Lu et al., [43]; Marton & Säljö, [44]). Vos et al., ([64]) and Chen, ([16]) found that students' intrinsic motivation and learning engagement positively related to their deep learning, separately. Moreover, Lei et al. ([39]) found that learning engagement was crucial for deep and productive learning. That is, students who had higher learning engagement might gain more learning achievement. In sum, students' intrinsic motivation level and learning engagement might serve as the essential indicators of their deep learning occurring (Biggs, [10]; Chen, [16]; Wang et al., [66]).

Cognitive load theory reveals the effect of cognitive load on the depth of learning processes and learning performance. When learners engage in deep learning occurs, if the information in the learning materials is overloaded or the complexity of the presentation or the difficulty of the learning task exceeds the learners' cognitive ability, a cognitive overload will occur, which may decrease learning performance.

Prior research work also found that intrinsic motivation and cognitive load should be considered and measured because they contribute differently to students' learning process and outcomes in multimedia environments (Lin et al., [41]). Therefore, the current study evaluated cognitive load as another essential indicator to measure deep learning performance. Furthermore, researchers also generally used knowledge retention and transfer tests to measure and assess students' learning outcomes (Schneider et al., [61]; Wang et al., [67]). In the research on the effects of cueing, Arslan-Ari et al., ([4]) used these tests to measure the impacts of prior knowledge and cues on students' learning and mental effort. Wang et al., ([67]) used the scores of retention and transfer tests to measure students' learning outcomes when exploring the effects of short instructional videos embedded clues. Ozcelik et al., ([55]) also combined the eye-tracking data and scores of retention and transfer tests to understand students' deep learning process and learning performance. Thus, students' retention and transfer test scores might be important indicators for evaluating students' deep learning outcomes in the current study.

According to the attention-guiding principle, also as known signal principle, cues are the information purposefully added in instructional materials to draw students' attention and provide visual scaffolding for better learning, such as highlighting the essential elements by coloring and text-picture (Juliane et al., [35]; Schneider et al., [61]; Wang et al., [67]). Cognitive Theory of Multimedia Learning holds that attention-guiding features (cues) should be used in instructional materials to promote the selection of essential learning information in the learning process. According to Mayer, ([46]), there are three types of cues in instructional animations: textual cues, visual cues, and combined textual-&-visual cues. Textual cues are composed of the explanatory text inserted into the learning materials, while visual cues mainly use direction arrow, color, "searchlight," and dynamic gestures. Visual and textual cues are often combined to guide attention to different learning materials (Xie et al., [70]). In addition, Mayer, ([47]) named the effective impact of these cues on improving learning outcomes as the cueing effect.

Studies showed that cues might positively affect learning (Arslan-Ari et al., [4]; Brasel & Gips, [13]; De Koning et al., [22]; Lin et al., [41]). Lin et al. (2011) added red arrows as visual cues to learning animations materials and proved that such signals could effectively improve learning effects and shorten learning time. Wang et al. ([67]) used eye-tracking approach and the retention and transfer tests to explore the functions and designs of added cues in short instructional videos. They found that the instructional videos added visual cues could promote deep cognitive processes and improve students' scores on knowledge retention and transfer tests. Yung and Paas ([75]) studied the effect of a teaching agent, a kind of cue, which prompts essential information in instruction animation about the cardiovascular system. Still, they did not find that the alert positively impacts students' learning and maintenance of cognitive load. A meta-analysis comprising 95 studies of cueing effect in learning with media showed that the cues benefit students' learning achievement (Schneider et al., [61]). However, Johnson et al. ([34]) found that multimedia materials with visual cues (arrow pointing) could speed up learning and reduce cognitive load but had no significant impact on academic performance. Plass et al. ([57]) did not find gray visuals and warm-color-cued visuals significantly affect an individual's cognitive load or intrinsic motivation. Lin et al. ([41]) also found that cueing had no significant effect on students' intrinsic motivation and cognitive load.

Although appropriate using cues (e.g., textual cues, visual cues, and combined textual-&-visual cues) could promote learners' attention, integration, and deep processing of important information, the empirical results of these studies still raise doubts about the effective occurrence of cues in promoting deep learning. Some researchers believe that these cues may effectively attract learners to pay attention to crucial learning information but could not enhance learners' engagement, making it challenging to promote the significant occurrence of deep learning (De Koning et al., [21]; Lin et al., [41]). For instance, van der Meij and de Jong, ([63]) argue that cues as relating representations on a surface level support learners, but gaining a deeper understanding for knowledge by reasoning with these cueing requires additional cognitive support. Therefore, incorporating cognitive strategies as instructional aids for cueing in learning materials should be considered to facilitate learners' mental model construction, which may be beneficial to foster deeper cognition and better learning outcomes. It is noted that self-explain prompts may be effective cognitive strategies to scaffold learners' engagement in cognitive processes and deep learning with instructional videos (Roy & Chi, [59]; Yeh et al., [73]; van der Meij et al., 2011).

Chi et al. ([19]) put forward the concept of self-explanation when he analyzed the difference between "excellent students" and "poor students" in the process of problem-solving in physical learning through experiments. Fonseca and Chi ([26]) believed that self-explanation is a constructive or in-depth cognitive activity in which learners respond effectively to prompts (additional non-knowledge information) to understand new content deeply or promote the meaning construction effectively. That is, self-explanation prompts are considered essential instructional aids to scaffold and encourage learners to engage in cognitive construction to facilitate deep and powerful learning through predication, reflection, reasoning, and revision of mental models (Chi, [18]; Chi et al., [20]; Roy & Chi, [59]). Therefore, the appropriate self-explanation prompts have the potential to trigger and improve learning engagement which enables learners to engage in deep learning processes. Previous research works, including a meta-analysis (Bisra et al., [11]), provide some positive evidence about the self-explanation prompts in enhancing the depth of learning processes and improving learning performance (Moreno & Mayer, [52]; van der Meij et al., 2011; Lin et al., [41]; Park et al., [56]).

Researchers pointed out that the types of self-explanation prompt, the presentation of prompting questions differently, may affect the inducing degree of self-explanation differently and thus affect learners' deep cognitive processing and deep learning to varying degrees (Atkinson et al., [5]; Kent et al., 2007; Yeh et al., [73]). For instance, Nokes et al. ([53]) designed gap-filling prompts and mental-model revision prompts to support students' problem-solving in physics subjects. Roy and Chi ([59]) also put forward several types of prompts, including open-ended prompt, focused prompt, scaffolded prompt, resource-based prompt, and menu-based prompt to engage students learning successfully from kinds of instructional materials. Notably, self-explaining is intrinsic cognitive construction activity (Roy & Chi, [59]), which means the prompting questions when to present and timely observing their impact on learners' meaningful understanding and knowledge construction is crucial.

Researchers generally refer to prompting questions given at the beginning of a related instruction as prediction prompts and questions that learners need to explain immediately after completing the instructional task as reflection prompts (Hajian et al., [29]; Lin et al., [41]). In the current study, prediction prompts were presented to students before delivering the instructional video content, which may help them activate their prior knowledge and ideas related to a specific concept and subsequently examine them in more detail (Hajian et al., [29]). While reflection prompts were presented after students had completed instructional video watching, which guides them to summarize and reflect on what they have learned or even work to resolve cognitive conflicts between old and new knowledge.

Previous researches confirmed that prediction prompts and reflection prompts positively affected students' cognitive processing, respectively (Hegarty et al., [31]; Moreno & Mayer, [52]). Besides, Lin et al., ([41]) investigated the impact of different types of self-explanation prompts (prediction prompts vs. reflection prompts vs. no prompts) and visual cueing on learning outcomes in a multimedia environment. They found that visual cueing significantly impacts learning outcome scores, but self-explanation prompts had no significant effect. It is noted that this result may be caused by the fact that multimedia learning materials are complex, dynamic, and transient (Wang et al., [67]), so it isn't easy to track and analyze how cues and self-explanation prompts affect the deep learning process. Much more research is needed to understand the effect of different types of cues and self-explanation prompts on cognitive processes and deep learning performance by analyzing multiple data, such as including learners' brain wave data and eye movement data.

This study investigates the effects of cues (visual vs. textual vs. combined textual-&-visual) and self-explaining prompts (prediction vs. reflection) used with the instructional videos on promoting students' deep learning. Specifically, this study addressed the following research questions (RQs):

• RQ1: Which type of cues (visual vs. textual vs. combined textual-&-visual) have more significant positive effects on deep learning performance?
• RQ2: Which type of self-explanation prompts (prediction vs. reflection) have more significant positive effects on deep learning performance?
• RQ3: Does any interaction effect exist for each deep learning performance measure among cues (visual, textual vs. combined textual-&-visual) by self-explanation prompts (prediction vs. reflection)?

This study manipulated two independent variables: cues (visual, textual vs. combined textual-&-visual) and self-explanation prompts (prediction vs. reflection). The different combinations of cues and self-explaining prompts were embedded into learning materials (an instructional video about ozone), respectively, to attract students' learning attraction, foster learning engagement, and stimulate the occurrence of deep learning activities. To examine how the combination of self-explaining prompts and cues affected the deep learning process and learning performance, we used four indicators as dependent variables to measure and interpret the effects: intrinsic motivation, learning engagement, learning outcomes (retention and transfer scores), and cognitive load.

N = 72 Chinese college students (50 females, 12 males; Mean [M] age = 23, Standard Deviation [SD] = 2.2, 18—25 years) took part in the study, which lasted about 40 min, respectively. The participants were majoring in educational technology or software engineering and had normal hearing and normal or corrected-to-normal vision. They had basic computer skills, but they were not familiar with instrument content. To ensure that participants take part seriously and obtain accurate experimental data, they will receive a small stipend after carefully completing the entire experiment.

This study used a 3 (visual vs. textual vs. combined textual-&-visual) × 2 (prediction vs. reflection) between-subjects design. The participants were randomly assigned in equal numbers (N = 12) to one of the six conditions (See Table 1).

Table 1 Six conditions and explanations

Visual Cues—Prediction Prompts, VC-PP Visual Cues—Reflection Prompts, VC-RP Textual cues—Prediction Prompts, TC-PP Textual cues—Reflection Prompts, TC-RP Combined Textual-&-Visual Cues—Prediction Prompts, CTVC-PP Combined Textual-&-Visual Cues—Reflection Prompts, CTVC-RP

The original experimental learning materials were selected from the short instructional video of Fuse School (www.fuseschool.org), lasting for 4 min and 10 s, which delivered an instructional unit about the ozone in the atmosphere. Specifically, the instructional video covered the following topics in a sequence: (a) the different characteristics of ozone in the stratosphere and troposphere, (b) the formation process of ozone in the stratosphere, (c) the uneven distribution of ozone in the atmosphere, (d) and the reason why most of the ozone in the troposphere becomes a pollutant. Considering the participants were native Chinese speakers, the English text elements of the original English instructional video were translated into Chinese by two graduate students proficient in English and dub the video into Chinese. Then, the different combinations of cues and self-explanation prompts were embedded into the six instructional animations, respectively (See Table 1). Cues appeared in the relevant animation area in each condition when the voiceover explained the eleven key instructional content. According to the topics in the teaching animation, the same type of self-explanation prompts was embedded at the beginning (prediction prompts) or end (reflection prompts) of each topic, a total of four in each condition.

The demographic (including gender, age, and major) and prior knowledge questionnaire designed in the study was mainly intended to obtain the basic information of participants and investigate participants' prior knowledge about the instructional content—ozone, to eliminate the interference of the participants with high prior knowledge on the experimental results. The prior knowledge questionnaire consisted of twenty basic questions related to ozone selected from an environmental protection knowledge contest, including five blank questions (e.g., the leading gases causing ozone layer destruction are____), eight multiple-choice questions (e.g., which of the following dates is international ozone layer protection day? A.6 December B.6 November C.6 October D.6 September), and seven judgment questions (e.g., ozone is a light blue gas with a particular smell.). Each question in the questionnaire was scored 0 for an incorrect answer or 5 for a correct answer, giving a total score of 100. If the test score were more than 65, the participant would be judged to know more about the instructional content and removed from the participant list. The internal consistency Cronbach's alpha for the questionnaire and the difficulty of the questionnaire were 0.72 and 0.40, respectively. There were no significant differences (F(1,5) = 0.026, p > 0.05) among the six condition groups, which included the VC-PP condition (M = 40.83, SD = 13.95), VC-RP condition (M = 40.00, SD = 10.00), TC-PP condition (M = 41.25, SD = 13.67), TC-RP condition (M = 40.42, SD = 12.87), CTVC-PP condition (M = 40.97, SD = 13.75) and CTVC-RP condition (M = 40.00, SD = 11.83).

We adapted a five-item instrument for measuring intrinsic motivation from the measurement developed by Ryan ([60]), including five sub-scales: interest, competence, value, effort, and pressure. The instrument was an eight-point Likert scale, with a total of five items, ranging from 1 ("Strongly disagree") to 8 ("Strongly agree") (See Table 2). The internal consistency of Cronbach's alpha for the instrument used in this study was 0.74, implying that it had good reliability.

Table 2 Intrinsic motivation items

Learning engagement refers to the "quantity and quality of mental resources directed at an object and the emotions and behaviors entailed" when individuals learn (Miller, [49], p. 31). Previous researchers have collected participants' physiological data by using an EEG detector or/an eye tracker to measure their learning engagement (Baceviciute et al., [6]; Dubovi, [24]; Wang et al., [65]). For example, Liu et al., ([42]) used the Tobii X120 eye tracker to collected students' gaze time data on learning materials to measure their learning engagement. Baceviciute et al., ([6]) incorporated eye-tracking and EEG to investigate participants' cognitive engagement during learning in a virtual reality environment. To measure participants' learning engagement in the current study, we applied a portable EEG detector, Neurosky mindwave mobile, and Tobii Pro X3-120 eye tracker to collect their brain wave data and eye movement data, respectively.

The portable EEG detector—including the sensor contacting the prefrontal lobe, the sensor reaching the earlobe, and the data processing chip—can collect and record four original brain wavebands on a real-time basis: alpha, beta, theta, delta with a sampling rate of 120 Hz and a signal accuracy of 0.25uV. In addition, we used the software kit Mindxp to convert the collected brain wave data into attention and meditation values ranging from 0 to 100 through eSense, a proprietary algorithm for representing mental states to analyze the collected brain wave data. It is noted that attention values reflect the degree of participants' current mental concentration level (cognitive engagement), while meditation values demonstrate the degree of mental relaxation level (emotional engagement) (Liu et al., [42]; Wang et al., [65]; Yang et al., [72]).

Tobii Pro X3-120 eye tracker with a sampling rate of 120 Hz was pasted under an HP display with a resolution of 1920 × 1080 to capture participants' eye movement data during the experiment. In general, Tobii studio 3.4.5 software was used to perform the calibration process and analyze the eye movement data, such as fixation allocation, fixation duration, fixation count, saccade, and a heatmap. This study marked the region of cues within the instructional video in each condition as the areas of interest (AOI). We explored two eye movement indicators: (a) the total fixation count of AOIs—the sum of fixation in the AOIs of participants from an experimental condition, reflecting participants' familiarity and attention degree to instructional materials. (b) The mean fixation duration of AOIs—the average time of all the fixation durations in the AOIs, reflecting the degree of participants' cognitive processing of learning materials. Previous studies have shown that these two eye movement indicators, which represent the participants' behaviors of selecting, attention, and processing information in learning material, can more accurately measure learning engagement (behavioral engagement) (Kaakinen, [36]; Wang et al., [67]; Yue et al., [74]).

In the study, we measured the learning outcomes in terms of the retention test and the transfer test, both developed by two experienced science instructors. The retention test mainly measured the extent of the participants' memory and mastery of declarative knowledge, explicitly delivered in instructional animation. The retention test questions are closely related to the instructional video content emphasized by the cues and/or self-explanation prompts, reflecting the impact on cognitive processes and depth of meaning construction (Ponce & Mayer, [58]; Wang et al., [67]). The test consisted of twenty questions, including five blank questions (e.g., ozone in the stratosphere can absorb __ from the sun and is the earth's protective umbrella.), eight multiple-choice questions (e.g., which of the following is the reason for less ozone distribution at the stratospheric tropospheric junction? A. no oxygen, B. six less oxygen, C. no ultraviolet rays, and D. six less ultraviolet rays), and seven judgment questions (e.g., ozone in the stratosphere can absorb 80% of the sun's ultraviolet rays and is the earth's protective umbrella.). Each question in the test was scored 0 for an incorrect answer or 5 for a correct answer, giving a total score of 100. In addition, the retention test's scoring procedures were consistent with those of the prior knowledge test, but the questions were different.

The transfer test measured the extent to that participants applied declarative knowledge to solve problems that were not expressly explained by instructional animation. The test consisted of five short questions—four questions related to self-explanation; another question needed participants to consider the instructional content before answering what they learned-closely associated with the self-explanation prompts. Two science instructors with deep understandings of ozone knowledge scored the transfer test according to the corresponding scoring standards. The scorers' consistency coefficient was above 0.88.

We adapted a five-item instrument for measuring cognitive load from the NASA-TLX developed by Hart and Staveland ([30]), including five sub-scales: interest, competence, value, effort, and pressure. The instrument was an eight-point Likert scale, with a total of five items ranging from 1 to 8 (See Table 3). The internal consistency of Cronbach's alpha for the instrument used in this study was 0.75, implying that it had good reliability.

Table 3 Cognitive load items

The experiment was carried out in a learning science laboratory of a Chinese college. At the beginning of the study, participants are required to sign a participation consent form. After each participant sat in front of a computer in the laboratory, the researchers briefed them on the experimental process. The experiment procedure included four steps in this study, as shown in Fig. 1.

Graph: Fig. 1Experiment design. Abbreviation: Visual Cues—Prediction Prompts, VC-PP; Visual Cues—Reflection Prompts, VC-RP; Textual Cues—Prediction Prompts, TC-PP; Textual Cues—Reflection Prompts, TC-RP; Combined Textual-&-Visual Cues—Prediction Prompts, CTVC-PP; Combined Textual-&-Visual Cues—Reflection Prompts, CTVC-RP

In the first step, the participants completed the demographic questionnaire and took the prior knowledge test. And then, they were randomly assigned to one of six experimental conditions. In the second step, the researchers helped participants wear EEG headsets and adjust their sitting posture using the 9-point calibration method. To ensure eye movement data capturing effectively, participants sited at approximately 65 cm from the display and did not move their heads as much as possible. In the third step, before carrying out the main experimental task, participants took a warm-up task unrelated to ozone in 5 min to familiarize themselves with the learning setting. In the fourth step, participants should complete the main experimental task with the instructional video within 10 min. During the main experiment, participants' mindwave data (attention and meditation data) and eye movement data (total fixation count and mean fixation duration data) were collected by an EEG headset and an eye tracker, respectively. In the fifth step, participants independently completed the retention test, transfer test, intrinsic motivation questionnaire, and cognitive load questionnaire. After submitting all the questionnaires, participants were thanked for their participation. Each participant took about 40 min to complete the whole experiment in this study.

We first checked the results for normality by computing the Shapiro–Wilk normality test and determined that the data sets were normally distributed (ps > 0.05). Then we applied the Statistical Package for Social Science 24.0 to process the experimental data—the usage of the two-way analysis of variance (ANOVA) analysis assessment of the potential effects of the cues and self-explanation prompts on the indicators of deep learning as follows: intrinsic motivation, learning engagement (the total fixation count and the mean fixation duration), learning outcomes (retention test scores and transfer test scores), and cognitive load. In addition, After the participants finished the learning activities, the participants' brainwave states of attention and meditation were coded, as shown in Table 4 and Table 5. According to Bakeman and Gottman ([8]) and Yang et al. ([72]), we used the sequential analysis method (SAM) to analyze the codes of attention and meditation.

Table 4 Attention coding

Table 5 Meditation coding

We used Levene's test for equality of variances to assess whether the homogeneity of variance on the intrinsic motivation scores of groups met homoscedasticity. After determining that Levene's test results for homogeneity of variance met homoscedasticity with F(5, 66) = 2.020, p = 0.087, a two-way ANOVA was conducted to examine the potential effects of cues and self-explanation prompts on intrinsic motivation. The results showed that neither of the two main effects was significant: for the main effect of cues, F(2, 66) = 0.739, p = 0.481, η2p = 0.022, and for the effect of self-explanation prompts, F(1, 66) = 0.657, p = 0.420, η2p = 0.010. In addition, the interaction effects between cues and self-explanation prompts on intrinsic motivation were not significant, F(2, 66) = 0.979, p = 0.381, η2p = 0.029.

Levene's test results on the total fixation count of groups met homoscedasticity, F(5, 66) = 1.369, p = 0.247. Then we conducted two-way ANOVA to examine the total fixation count of AOIs and the mean fixation duration of AOIs. The results indicated that there were neither of the two main effects on total fixation count was significant: for the main effect of cues, F(2, 66) = 0.274, p = 0.761, η2p = 0.008, and for the main effect of self-explanation prompts, F(1, 66) = 0.001, p = 0.977, η2p = 0.001. Moreover, the interaction effects between cues and self-explanation prompts on total fixation count were significant, F(2, 66) = 3.907, p = 0.025, η2p = 0.106). We follow-up performed the simple effects and found that the participants from the VC-PP condition (M = 109.83, SD = 23.74) have a significantly higher total fixation count (ps < 0.05) than those from the other conditions—VC-RP condition (M = 83.58, SD = 27.89), TC-PP condition (M = 86.75, SD = 26.75), CTVC-PP condition (M = 90.00, SD = 20.04), TC-RP condition (M = 97.25, SD = 23.52), and CTVC-RP condition (M = 105.17, SD = 41.60), respectively (See Fig. 2).

Graph: Fig. 2Interaction between self-explanation prompts and visual cues on total fixation count and mean fixation duration. Abbreviation: Visual Cues, VC; Textual Cues, TC; Combined Textual-&-Visual Cues, CTVC; Prediction Prompts, PP; Reflection Prompts, RP

After Levene's test results on the mean fixation duration of groups met homoscedasticity with F(5, 66) = 2.029, p = 0.086, we conducted a two-way ANOVA. The results showed that there were main effects of cues, F(2, 66) = 10.939, p < 0.001, η2p = 0.249, and self-explanation prompts, F(1, 66) = 11.256, p = 0.001, η2p = 0.146, on mean fixation duration. A follow-up Least Significant Difference (LSD) test found the results as follows. (a) The mean fixation duration of participants from the VC condition (M = 0.31, SD = 0.13) was significantly longer than those from the TC condition (M = 0.25, SD = 0.10), and CTVC condition (M = 0.19, SD = 0.06). (b) The mean fixation duration of participants from the TC condition (M = 0.25, SD = 0.10) was significantly longer than those from the CTVC condition (M = 0.19, SD = 0.06), (See Fig. 3). (c) The mean fixation duration of participants from the RP condition (M = 0.29, SD = 0.13) was significantly longer than those from the PP condition (M = 0.21, SD = 0.07).

Graph: Fig. 3Mean of mean fixation duration of three cue conditions. Abbreviation: Visual Cues, VC; Textual Cues, TC; Combined Textual-&-Visual Cues, CTVC

Moreover, the interaction effects between cues and self-explanation prompts on the total fixation count were significant, F(2, 66) = 3.907, p = 0.025, η2p = 0.106. We follow-up performed the simple effects and found that the participants from the VC-RP condition (M = 0.39, SD = 0.12) have a significantly longer mean fixation duration than those from the other conditions (See Fig. 2).

This study used the EEG to collect and record participants' mindwave data (attention and meditation) during the experimental learning activities. According to Yang et al. ([72]) and Hou et al. ([32]), the sequential analysis method could analyze the message codes—participants' attention and meditation—and describe the code sequences with significance (Z) of the Z factor binomial test. Each node represented a code in a message code transfer diagram, and lines indicated the significance levels where the significant Z > 1.96 (Bakeman & Gottman, [8]), and the arrow pointed to transfer directions. To observe the significant code transfer visually, we drew the transition diagrams of attention and meditation, respectively.

As shown in Fig. 4, participants from the TC-PP condition and TC-RP condition continuously stayed in "high level of attention" (A4) and "very high level of attention" (A5) states with the significant Z = 3.61 and 2.50, respectively. It is noted that participants from the TC-RP condition did not have a "very low level of attention" (A1) states. While participants from the other four conditions all stayed in had significantly stable states of "very low level of attention" (A1) and "low level of attention" (A2). Especially, participants from the VC-PP condition and CTVC-RP condition continuously stayed in "very low level of attention" (A1) states with the significant Z = 3.61 and 3.05, respectively. That is, the mind wandered continuously appeared during their learning activities. In sum, the participants from the TC-RP condition easily raised the high-level attention and could constantly maintain it.

Graph: Fig. 4The attention transformation diagram of the six conditions. Abbreviation: Visual Cues—Prediction Prompts, VC-PP; Visual Cues—Reflection Prompts, VC-RP; Textual Cues—Prediction Prompts, TC-PP; Textual Cues—Reflection Prompts, TC-RP; Combined Textual-&-Visual Cues—Prediction Prompts, CTVC-PP; Combined Textual-&-Visual Cues—Reflection Prompts, CTVC-RP

Brainwave meditation can represent participants' level of mental calmness or relaxation. When the meditation value increased, the participant was in a relaxed state, and when it decreased, the participant was in a stressed state. It is noted that the meditation value is too high (high level of relaxation) or too low (high level of stress) is not necessarily conducive to the learning process. The meditation states moderate tension, relaxation, or even neutral are more beneficial to the participants engaged in the learning activities.

As shown in Fig. 5, the mediation of participants from the TC-PP condition and TC-RP condition more concentrated the transitions between M3—the natural level of meditation—and M4—the high level of meditation, which showed that the participants could maintain moderate relaxation states. Especially, the participants from the TC-RP condition did not have a "very high level of mediation" (M5) states, which represented they had the most stable mediation, compared to other participants. The participants from the condition of VC-PP, VC-RP, and CTVC-RP have the concentrated transitions between M4—the high level of meditation—and M5—the very high level of meditation. Notably, participants from the condition of CTVC-PP continuously stayed in "very high level of meditation" (M5) states with the significant Z = 2.66. That is, the mind wandered might constantly appear at some stage in their learning activities. In sum, the participants from TC-PP and TC-RP conditions easily raised and kept moderate meditation transition states—the natural state or a high level of relaxation.

Graph: Fig. 5The meditation transformation diagram of the six conditions. Abbreviation: Visual Cues—Prediction Prompts, VC-PP; Visual Cues—Reflection Prompts, VC-RP; Textual Cues—Prediction Prompts, TC-PP; Textual Cues—Reflection Prompts, TC-RP; Combined Textual-&-Visual Cues—Prediction Prompts, CTVC-PP; Combined Textual-&-Visual Cues—Reflection Prompts, CTVC-RP

Levene's test results on the retention and transfer scores of groups met homoscedasticity with F(5, 66) = 0.991, p = 0.430, and with F(5, 66) = 0.532, p = 0.752, respectively. Then two-way ANOVA analyses were conducted to evaluate the effects of cues and self-explanation prompts on the retention and transfer test scores.

The results indicated that there were main effects of cues, F(2, 66) = 5.012, p = 0.009, η2p = 0.132, on the retention test scores. A follow-up LSD test found that a) participants from TC condition (M = 59.58, SD = 10.73) scored significantly higher than those from VC condition (M = 49.79, SD = 8.91), and CTVC condition (M = 51.88, SD = 13.42), respectively (see Fig. 6). However, the main effect of self-explanation prompts, F(1, 66) = 3.518, p = 0.30, η2p = 0.016, and the interaction between cues and prompts, F(2, 66) = 0.248, p = 0.781, η2p = 0.007, were insignificant.

Graph: Fig. 6Mean of the retention test and the transfer test of three cue conditions. Abbreviation: Visual Cues, VC; Textual Cues, TC; Combined Textual-&-Visual Cues, CTVC

A two-way ANOVA results indicated that there were main effects of cues, F(2, 66) = 3.527, p = 0.035, η2p = 0.097, on the transfer test scores. A follow-up LSD test found that participants from TC condition (M = 46.27, SD = 13.61) scored significantly higher (ps < 0.05) than those from CTVC condition (M = 35.69, SD = 14.74). In addition, the interaction effects between cues and self-explanation prompts on the transfer test score were significant, F(2, 66) = 4.960, p = 0.010, η2p = 0.131 (See Fig. 7). We follow-up performed the simple effects and found the results that the participants from VC-PP condition (M = 49.96, SD = 15.06) and TC-RP condition (M = 50.63, SD = 15.61) scored significantly higher (ps < 0.05) than those from VC-RP condition (M = 34.50, SD = 11.94), CTVC-RP (M = 37.29, SD = 14.13), TC-PP (M = 41.92, SD = 10.12), and CTVC-PP (M = 34.08, SD = 15.78), respectively.

Graph: Fig. 7Interaction between two prompts and visual cues on the transfer test. Abbreviation: Visual Cues, VC; Textual Cues, TC; Combined Textual-&-Visual Cues, CTVC; Reflection Prompts, RP; Prediction Prompts, PP

However, the main effect of self-explanation prompts on the transfer test scores were insignificant, F(1, 66) = 0.129, p = 0.720, η2p = 0.002.

Levene's test results on the cognitive load scores of groups met homoscedasticity, F(5, 66) = 1.521, p = 0.195. Then we conducted a two-way ANOVA to examine the potential effects of cues and self-explanation prompts on cognitive load. There was a significant main effect of self-explanation prompts, F(2, 66) = 7.595, p = 0.008, η2p = 0.103. Participants assigned to the PP conditions (M = 15.42, SD = 4.67) scored significantly lower than those who were assigned to the RP conditions (M = 18.86, SD = 6.01). However, the main effect of cues, F(1, 66) = 2.351, p = 0.103, η2p = 0.066), and the interaction between cues and prompts, F(2, 66) = 0.730, p = 0.486, η2p = 0.22, were insignificant.

The purpose of this study was to measure the effect of different types of cues (visual vs. textual vs. combined textual-&-visual) and self-explanation prompts (prediction vs. reflection) in instructional videos on participants' deep learning performance. The study adopted multiple data metrics to measure and understand the participants' deep learning performance levels through four indicators—intrinsic motivation, learning engagement, learning outcomes, and cognitive load. Moreover, to measure their learning engagement—attention, meditation, the total fixation, and the mean fixation duration—in the learning process, we used a portable EEG detector and an eye tracker to collect their brain wave data and eye movement data, respectively. The results revealed that the textual cues significantly facilitated learning outcomes and learning engagement—attention-, while the reflection prompts significantly affected learning engagement—the mean fixation duration—and cognitive load. Notably, the combination of textual cues and reflection prompts (TC-RP) and the combination of visual cues and prediction prompts (VC-PP), respectively, allowed the participants to focus and engage in the video learning process more deeply, resulting in a significantly higher learning outcome than their peers from other conditions.

The current study results revealed that using the textual cues significantly facilitated learning outcomes—retention test and transfer test scores—in the instructional video learning activities. The beneficial effect of textual cues on learning outcomes is consistent with some previous studies' results (Boucheix & Lowe, [12]; Canham & Hegarty, [14]; Miller et al., [50]; Stark et al., [62]; Wang et al., [68]). For instance, a multimedia courseware design study also showed that the effect of textual cues on learning outcomes was significant (Wang et al., [68]). However, some previous studies found that visual and textual-&-visual cues have more positive effects than textual cues on students' learning outcomes which were inconsistent with our findings (Boucheix & Lowe, [12]; Wang et al., [67]). A possible reason is that the added cueing text combined with the animations, pictures, and voice-overs related to the learning content in the instructional videos foster the cognitive processes—the selection, organization, and integration of learning information—which was beneficial to the occurrence of deep learning. The Cognitive Theory of Multimedia Learning (Mayer, [46], [47]) provided support to our findings and the possible consequence that the text entering the working memory through an individual's visual channel will combine with the visual representation (e.g., picture, animation, video) and the entering sound (e.g., voice-over) through the auditory channel to form a "phrase," which help learners to process the learning materials deeply and significantly improve the learning effect. In addition, the results from the EEG data in the current study showed that the participants from the textual cues condition maintained a higher level of attention (A 4 and A5), indicating that they engaged in continuous deep cognitive processing, which was beneficial for better learning outcomes.

Our results did not show that different cues significantly impacted intrinsic motivation and cognitive load, consistent with the previous experimental findings—cueing had no obvious direct effect on intrinsic motivation and cognitive load. (Arslan, [3]; Lin et al., [41]). For the insignificant difference between different cues on intrinsic motivation, the possible reason is that volunteer participants from different majors might not be interested and motivated towards learning the content (Arslan, [3]). However, we did not find participants have the insignificant difference of cueing effects and the prompting-by-cueing interaction on intrinsic motivation, which also provided evidence for our above explanation to the results. In terms of cognitive load, the findings are in line with previous studies with the possible reason that the short learning materials were not easy to generate cognitive load, especially for the 4 min—instructional video (De Koning et al., [23]; Kriz & Hegarty, [38]; Lin et al., 2011; Mautone & Mayer, [45]; Yang, [71]). Another possible reason was the participants easily ignored the static embedded cues appearing several times in instructional videos, which could not increase their cognitive load.

The current study results revealed that the reflection prompts significantly affected learning engagement—the mean fixation duration—and cognitive load, while no significant impact on other indicators of deep learning performance. The mean fixation duration was the crucial positive indicator for measuring the engagement in selecting information during learning processes (Wang et al., [67]). In the current study, the positive effect of reflection prompts on learning engagement is consistent with the findings of some empirical studies (Chen, [15]; Hung et al., [33]). A standpoint of the feedback principle of multimedia learning—providing feedback to guide learners to engage a deeper understanding of learning materials—also supports the benefits of reflection prompts on learning engagement. Moreover, according to the opinion voiced by Hung et al. ([33]), the reflection prompts were beneficial to direct learners to pay more attention deeply in understanding material and promote deep cognitive processes. In addition, compared with the participants using predictive prompts, those who use the reflection prompts have significantly higher learning engagement in the cognitive process, being easy to result in a higher cognitive load. The Cognitive Theory of Multimedia Learning also supports the findings that when learners deeply engage in selecting and integrating information in the cognitive processes during multimedia learning, their germane cognitive load will be increased (Mayer, [47]).

The results showed the insignificant benefits difference between the different self-explanation prompts on learning outcome, consistent with some empirical studies' findings (De Koning et al., [22]; Gerjets et al., [27]; Lin et al., [41]; Moreno & Mayer, [51]). Lin et al. ([41]) hold a possible explanation for the similar findings that the participants might not do any self-explanation during learning actives. In the current study, however, the different types of self-explanation prompts significantly differed on the learning engagement—the mean fixation duration, indicating the participants had used the reflection prompts during their learning activities. Therefore, the possible explanation for the insignificant learning outcome was that the negative effect of higher cognitive load offsets the positive effect of learning engagement on learning outcome.

The current study found significant interaction effects among different cues and self-explanation prompts on the transfer test scores and learning engagement. Specifically, the combination of textual cues and reflection prompts (TC-RP) and the combination of visual cues and prediction prompts (VC-PP) were significantly beneficial to participants' transfer test scores and learning engagement (e.g., total fixation count and attention, respectively). In the current study, the prediction questions related to the instruction before the video learning might activate participants' relevant prior knowledge and guide them purposefully to select and analyze the information related to the predicted question or the prior knowledge (Lin et al., [41]). What's more, during the participants selecting and analyzing instructional video information, the visual cues could more directly and vividly arouse participants' cognitive attention in purposeful learning information related to the prompting question, which could help them establish a deeper connection between prior and new knowledge, and facilitated the knowledge transfer. In the current study, the participants from the VC-PP condition have a significantly higher total fixation count than those in the other conditions, which also possibly explained that they were engaging in selecting the information related to the predicted prompts with the guidance of visual cues.

On the other hand, the participants from the TC-RP condition were less purposeful during the instructional video learning than those who were delivered the prediction prompts. Therefore, they might have to pay attention to and fully understand the video information during their video learning activities. Notably, the textual cues—using colors in texts or different intonations—could draw the participants' attention to the key locations where the essential learning information was, helping them deeply analyze and understand the information (Wang et al., [67]). So that could explain why the participants from the TC-RP condition continuously stayed in "very high level of attention" (A5) states and the most stable mediation—maintain the moderate relaxation states. What's more, Hung et al. ([33]) and Chen ([15]) pointed out that "reflection prompts can guide students to think deeply, so as to improve their learning performances via connecting their prior concepts with new ideas." That is, in the current study, the reflection prompts provided the participants who used the textual cues crucial opportunities to reorganize and integrate information, facilitated them to deeply understand the relationship between prior and new knowledge, which was helpful for their significant performance in the transfer test.

This study examined the effects of different types of cues and self-explanation prompts on students' deep learning performance in instructional videos. Although we collected and analyzed multiple data about the deep learning performance in the current study, there are several limitations. First, the participants were from a particular discipline in a normal university in China, and the number of female participants (50) was larger than that of male participants (12). In future, researchers who plan to repeat the experiment should consider the possibility that participants from other disciplines and/or cultural backgrounds and balance the number of female and male participants. Second, since the wearing EEG headset is unsuitable for long study periods and asking participants not to move their heads as much as possible, the instructional videos used in the study were about 4 min, which might affect the experimental results. We suggest that future studies should properly extend the duration of the instructional videos and use the more advanced EEG headset. Third, our study mainly focused on the four indicators (intrinsic motivation, learning engagement, learning outcomes, and cognitive load) that may not include others equally important for studying deep learning. Future researchers should use more accurate multimodal data analysis methods to explore the occurrence of deep learning and evaluate it from multi-dimensional indicators, such as participants' emotions, behavior, presence, and social interactions.

The results of this study could provide some implications for instructors and learning materials developers. On the one hand, when instructors or developers design short instructional videos for college students, suggesting that using the textual cues to attract college students' attention engages them in the cognitive processing deeply and facilitates their learning outcomes. On the other hand, when instructors or developers consider combining the cues and self-explanation prompts in a video learning setting, they could consider applying the combination of text-cued and reflection prompts (TC-RP) or the combination of visual-cued and prediction prompts (VC-PP) to foster students' video learning engagement and outcomes. In addition, we recommend applying the proper cognitive tools or strategies in the video learning setting to motivate students to engage in learning more deeply and decrease their cognitive load.

This study was supported by the Jiangsu Social Science Foundation Youth Project (20JYC002) and the National Natural Science Foundation of China (62077030).

The authors declare that data associated with this paper will be made available upon reasonable request.

The authors declare that there is no conflict of interest.

All the participants of this experiment were informed of the objectives of the same. All data have been anonymised, guaranteeing the privacy of the participants.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.