Date of Degree
Reading, Eye Tracking, Comprehension
With the growing trend of using multimedia platforms such as YouTube to facilitate storytelling, understanding how and when to integrate text with visuals would benefit both the creators of these platforms and the young readers viewing them. The current study examined the effect of orthography on vocabulary acquisition and narrative comprehension in young readers (children in 2nd and 3rd grade, ages 6-9) during a computer-based storytelling task. We aimed to determine if having text available during storytelling benefits readers as predicted by Perfetti’s Lexical Quality Hypothesis (Perfetti & Hart, 2002), or hampers learning as predicted by Mayer’s Redundancy Effect (Moreno & Mayer, 2002). The dissertation also explored if having access to the text would affect children’s eye gaze patterns while viewing the story, which had been found during the pilot study. It was hypothesized that the group who had access to their native English text would perform better in measures of comprehension and vocabulary knowledge and that they would show a different eye gaze pattern during the computer-based storytelling task than the group presented with Greek symbols in place of English.
Children completed a battery of pretests to assess reading level and working memory ability and then were randomly assigned to conditions where picture books displayed either the story text in English (meaningful condition) or in Greek symbols (meaningless text condition). They then watched a PowerPoint presentation featuring Mercer Mayer’s “Frog, where are you?” (Mayer, 1969) while having their eye gaze monitored using Tobii 4C eye tracker. After viewing the presentation, the children retold the story and answered ten comprehension questions, assessing both direct and inferential comprehension. They then completed a vocabulary knowledge scale that used words from the stories to assess vocabulary comprehension, as well as a cloze task created by the experimenter using target words from the story.
Results found that the presence of text did not affect comprehension or vocabulary acquisition, as both groups had similar scores across measures. However, the study found significant differences in eye gaze patterns between the groups. In both conditions participants spent significantly more time viewing the text during the beginning of the story and then reduced their dwell time during the middle and the end. Participants in both conditions also spent a higher percentage of dwell time on the illustrations during the beginning and middle of the story before reducing their focus at the end. However, when compared to participants exposed to the Greek symbols, participants with access to their native text spent significantly less time viewing the story pictures overall.
Results also found that for each page of the story, all participants focused on the text more before and during the playing of the recorded audio narration than after the narration was completed. An interaction between condition and the time stage of narration on dwell time suggested that narration affected participants differently depending on whether they had access to the meaningful text. Specifically, the participants with access to their native English text showed variation in their dwell times between the three different time stages: before, during, and after narration, which was not seen in the group who saw the Greek symbols.
While the study needs to be replicated to confirm results due to limitations caused by sample size and data loss in eye tracking, the present findings could help educators better integrate multimedia story telling within their classrooms by designing the activities to better match their students’ attentional and cognitive needs.
Ullrich, Nicholas J. III, "The Influence of Text on Coherence of Story Retells, Reading Comprehension, Vocabulary Acquisition, and Eye Gaze: A Computer-Based Story Telling Task with Eye Tracking" (2021). CUNY Academic Works.