Children’s Picturebook Goes Digital: Implications on Cognition

The objects of analysis in this article are digital picturebooks, which may be called e-picturebooks. This research contributes to a definition of e-picturebook as a distinct storytelling experience from printed picturebook, introducing distributed cognition as a new theoretical perspective for the analysis of the phenomenon. This perspective emphasizes cognitive systems related to specific features of this category of digital book. In this sense, picturebook and epicturebook are defined as cognitive artifacts that constrain storytelling tasks very differently from each other, not only because of their constitutive features but also because of the cognitive systems involved in the understanding of stories in each medium.

The objects of analysis in this article are digital picturebooks, which may be called e-picturebooks. This research contributes to a definition of e-picturebook as a distinct storytelling experience from printed picturebook, introducing distributed cognition as a new theoretical perspective for the analysis of the phenomenon. This perspective emphasizes cognitive systems related to specific features of this category of digital book. In this sense, picturebook and epicturebook are defined as cognitive artifacts that constrain storytelling tasks very differently from each other, not only because of their constitutive features but also because of the cognitive systems involved in the understanding of stories in each medium.

K E Y W O R D S
picturebook; book-app; application; storytelling; distributed cognition; cognitive artifacts.

P A L A V R A S -C H A V E
livro ilustrado; book-app; aplicação; narração; cognição distribuída; artefactos cognitivos.

I.WHAT IS AN E-PICTUREBOOK?
icturebook is often used as a generic term to categorize various types of publications which contain texts and images. However, according to the picturebook theory, that is a category that designates quite specific artifacts. Picturebook is indeed a printed narrative artifact consisting of "two levels of communication, the visual and the verbal" -specifically, illustration and verbal text (Nikolajeva and Scott, 2001: 1). But, it is the way the visual and verbal elements relate each other on the medium that will determine its classification as a picturebook.
In summary, Maria Nikolajeva and Carole Scott argue that those relations present states of redundancy or "different degrees of 'dependence' and 'enhancement' (…) as well as different natures of dependence and/or enhancement" (8). The interaction possibilities of those distinct types of signs in doublespread pages (sequence of two facing pages) are quite diverse and were defined by the researchers as results of the tension generated by differences of functions that text and illustration usually have. According to them, the function of text is mainly to narrate, while illustration describes or represents. Nikolajeva and Scott also emphasize that the texts are linear, in general, as opposed to illustrations, which do not have imposed instructions on how to read them.
Although the difference between verbal text and illustration is apparent, the dialectical relationship between those elements mutually reinforces or modifies them. The verbal text has its gaps and so does the visual. "Words and images can fill each other's gaps, wholly or partially. But they can also leave gaps for the reader/viewer to fill" (Nikolajeva and Scott, 2001: 2). Briefly, this is how picturebooks work for people to understand their stories.
During the last decade, the invention of mobile computing devices such as tablets and smartphones led to the development of digital reading experiences including even more varied features than the e-books designed for the first ereaders. One of those is the book-app. Currently, it is one of the formats that best expands the possibilities of digital reading because it is an application programmed for an experience largely influenced by interactive multimedia elements and different ways of exploring content. This is possible because those applications are designed to fully exploit the features of the devices and operational systems for which they have been programmed (mainly iOS and Android).
In the article entitled "From a Click to a Gesture," Ana Lúcia Pinto, Nelson Zagalo and Eduarda Coquet suggest the term e-picturebook to designate a specific category of book-apps: the interactive digital picturebooks for children. They propose "the designation of e-picturebook for these artefacts, establishing links with the original artifact, from which it inherits its main characteristics," that is, the picturebook (225). The researchers confirm that "e-picturebooks are anchored in picturebooks, in how they use the image and verbal text and in the way both establish, between the used mediums, semantic relationships" (227). Betty Sargeant also published a paper about theories and methodologies that are applied to the production of printed picturebooks and would also be relevant for the design and analysis of book-apps (Sargeant, 2013). Those studies have demonstrated similarities between printed and digital artifacts.
However, some features fairly differentiate e-picturebooks from the printed ones. In order to specify such features, a survey 1 was carried out, based on the work of some researchers who analyzed the book-app format. In addition, an analysis 2 of the book-apps from the Bologna Ragazzi Digital Award was also carried out. This very important world award has been offered annually to the best children's book-apps since the 2012 edition of the Bologna Children's Book Fair. Our examples included the winners and honorable mentions from 2012 until 2016 in the fiction category, totaling fifteen book-apps. Those book-apps are epicturebooks, according to Pinto's et al. definition.
The main characteristics found in the bibliographic survey and the analysis of book-apps were: (i) multimedia resources; (ii) multiple ways of presenting the verbal text; (iii) real-time interactivity; (iv) gamification; (v) new forms of content organization. Those features may be observed with greater or lesser prominence in different book-apps.

II.INTRODUCING THE CONCEPT OF COGNITIVE ARTIFACT
Human cognition may be defined as the set of all processes related to the acquisition and processing of information and knowledge in general -"including attending, remembering, and reasoning; also, the content of the processes, such as concepts and memories" (APA, 2002). Those processes are involved in the understanding of stories. In order to investigate the cognitive impact of book-app's digital features on the understanding of stories, we suggest a theoretical framework called situated and distributed cognition.
Distributed cognition is a hybrid approach derived from cognitive science, cognitive anthropology, and social sciences. It was developed by Edwin Hutchins and his fellows from the University of California in the late 1980s (Rogers, 1997).
This approach re-evaluates cognitive phenomena paying special attention to the relations between human cognition and the physical and cultural environment by analyzing the variety of mechanisms capable of participating in those processes. Unlike more traditional approaches of cognitive science that argue that the cognitive processes are always happening just inside individuals' brains, distributed cognition has challenged the relevance of skin and skull as clear spatial boundaries of mental activity. In other words, Hutchins asserts that human cognition may be understood as a distributed phenomenon in which the interaction between various individuals, technological artifacts, and social constructs are indispensable parts (Hutchins, 2001).
Observing various human activities and applying the idea of cognition as a distributed phenomenon, Edwin Hutchins found three distinct types of distribution: (i) through the members of a social group; (ii) through cooperation between internal and external structures; and (iii) through time. Since the purpose of this paper lies specifically in the relation between certain artifacts and mental processes isolated in time, we are focused on the second case of cognitive distribution.
The so-called external structure refers to the material world. Tools as diverse as pencil and paper, calculators, computers, calendars, maps, mathematical notations, and many others are considered non-biological artifacts that enable cognitive processes outside the boundaries of the individual's body. In this sense, they may be called "cognitive artifacts" (Norman, 1993: 47). Some theoretical approaches define cognitive artifacts as a means of amplifying the cognition of the user. Donald Norman argues that the power of the isolated mind is overestimated and skills such as memory and reasoning are restricted without external reinforcements. A greater potential of the mind would come by the invention of external artifacts that increase cognitive abilities (3).
The potential of the material environment to 'support' memory is an idea quite accepted in common sense, but sometimes such examples do not seem to amplify one's memory capacity in fact. As Michael Cole and Peg Griffin argue, an individual who took a note on a piece of paper to remember something did not have his memory amplified. This individual was actually using a different set of functional skills thanks to the cultural technologies applied to the task of remembering. The use of pencil and paper "restructured the activity so that some index of productivity was larger" (Cole and Griffin, 1980: 350). In this sense, pencil and paper, as cognitive artifacts, are "involved in a process of organizing functional skills into cognitive functional systems" different from those systems triggered on recall process by the mind alone (Hutchins, 2001(Hutchins, : 2070. Therefore, a way of harnessing the brain's capabilities and minimizing its weaknesses may be to combine neural systems with various tools and external structures, forming complex cycles between biological processes and others delegated to the external technological environment (source of complementary capabilities). Technologies can transform complex problems by absorbing tasks that would be exhausting to the brain. Those cognitive artifacts act on problem-solving efficiency, reducing the cognitive cost of an operation (when we use a calendar to find out which day will be the second Sunday of the next month); increasing the precision of a task (when we use a scale to check the weight of an object); or enabling a totally different set of functional skills to perform a task that would be impossible to the brain alone (when we use mathematical formulas to perform complex calculations).
Some tools are better integrated with human activities than others, so they may be called "transparent technologies" (Clark, 2003: 28). They are so well fitted to life that they do not become objects of our attention when we use them. In contrast, opaque technology is one that is always the focus of user attention. Usually, this occurs because those tools require skills that "do not come naturally to the biological organism" (37). For a person who performs a task with the help of opaque technology, the activity consists of controlling the tool successfully. Using transparent technology, the conscious agent sees "through the tool" and pays attention to the real task (38). The blind man's stick may work as transparently as some well-used and integrated high-tech devices.
Technologies considered opaque can become transparent. The process may consist of just a simple training to learn how to use a tool, or systematic training when extensive processes are needed to match biological organisms to tools. Transparency and opacity are not related to the complexity of a tool, but to its suitability to the human biological system.
Although this research deals with recent technological developments in the field of digital books, socio-technological matrices including biological organisms are not something contemporary, and theories about cognitive artifacts present examples from technologies much older than the previously mentioned pencil and paper. Looking back at the history of our cognitive technologies we perceive the starting point in speech and counting, then the written text and numerals, early printing, the movable type printing, and now we have multimedia resources encoded in digital formats. Such technologies constitute "a cascade of 'mindware upgrades': cognitive upheavals in which the effective architecture of the human mind is altered and transformed" (Clark, 2003: 4). However, Andy Clark emphasizes that "upgrades, as we all know, can be mixed blessings. Every new capacity brings new limits and demands" (10). Cognitive artifacts may be described as having the power both to create means for solving problems and to create new problems. That is, they create new structures to solve issues that, on the other hand, open up possibilities to the creation of new problem-spaces. When these two aspects are taken into account, cognitive artifacts can be seen as shaping cognition itself: endowing it with both capabilities and needs, creating increasingly specialized tools to deal with increasingly specialized tasks.
This fusion between mental capabilities and external artifacts is possible thanks to the biological condition that gives human beings an unusual degree of neural plasticity. Constructivist neuroscientists argue that the human cortex has mechanisms of neural growth that give it the characteristic of plasticity, that is, of being able to be "shaped and sculpted by the problems, resources, and opportunities encountered during postnatal and lifetime learning" (Clark, 2003: 84). In addition to that, other research has drawn attention to childhood as a period in which successive cultural structures can "change the dynamics of the cognitive system in a way that opens up new cognitive possibilities" (Griffiths and Stotz, 2000: 1-2).
The book has historically been one of the most successful ways of producing and disseminating knowledge, including through fiction. Especially for children, books occupy an important position because the consequences of reading extend beyond the task of understanding stories. They "carry profound implications for the development of a wide range of cognitive capabilities" (Cunningham and Stanovich, 2001: 137). Donald Norman argues that the "powers of cognition come from abstraction and representation: the ability to represent perceptions, experiences, and thoughts in some medium other than that in which they have occurred" (Norman, 1993: 47). We represent by using symbols, sounds, gestures, images, everything that can mean objects, concepts, sensations. The creation of representations allows human beings to deal with events and things that are not present in space and time; which also means that it allows one to relate to what does not even exist, as fictional worlds.
Best-selling author Reif Larsen, who has written both conventional books and digital storytelling experiences, made an emblematic account related to the concept of cognitive artifact in an interview with Richard Lea. A self-proclaimed believer in constraints as catalysts for creativity, Larsen argues that the emergence of unforeseen problems during the conception of a literary work guides the solutions to a more innovative direction. When the prototype for his digital storytelling experience -Entrances & Exits -was up and running, the story began to teach him "how it wanted to be told. It was creating a grammar of story all on its own" (Lea, 2016). According to the author, to create an innovative experience of storytelling it is necessary to use all the old tricks, but also let the platform guide the story a bit.
Both reading and story-making (regardless of whether they are printed or digital) require complex cognitive systems. Therefore, applying the cognitive artifact theory previously presented, and taking into account the well-marked material difference between picturebook and e-picturebook, it is possible that those storytelling experiences relate to two different cognitive systems operating in each kind of book. What does it mean for the task of understanding stories?

III.RESEARCH ABOUT STORIES UNDERSTANDING IN E-PICTUREBOOKS
Illustrations undergo drastic changes in e-picturebooks because features like motion in animations, sound effects, and music are added to them. In a survey conducted over a decade ago, Maria DeJong and Adriana Bus analyzed the multimedia features of storytelling experiences produced for the desktop (the iPad had not yet been created). They concluded that such features did not cooperate for the understanding of the stories (DeJong and Bus, 2003). However, the technological development of digital devices and storytelling experiences has led to different conclusions.
In a recent paper, Adriana Bus and Zsofia Takacs presented the results of an empirical study on the visual processing of static and animated illustrations by 4-6-year-old children (Takacs and Bus, 2016). The group, consisting of children from three public schools in the Netherlands, was subjected to storytelling experiences containing narrated verbal text accompanied by static or animated illustrations at different stages of the experiment. Each story was presented three times while children's eye movements were recorded by an eye-tracker.
Takacs and Bus based their research on dual coding theory as the starting point. According to that theory, when two sources of information (such as narration and animations) are not incompatible, they "can be processed simultaneously without causing cognitive overload" (Takacs and Bus, 2016: 2). Takacs and Bus's hypothesis was that the movement included on illustrations in digital picturebooks could guide children's visual attention, helping to find an important detail on the illustration. That should result in longer fixations on important parts of the image, combining them with verbal text narrated more effectively than static illustrations in printed books.
The results of the eye-tracker tests showed that the movements really directed the children's attention. The researchers also found that children remembered significantly more words from animated content than from the static version. Sulzby's scale of reading states that "the amount of verbally reproduced text is an important indicator of children's level of story comprehension" (Takacs and Bus, 2016: 5). Therefore, the results confirm the hypothesis that animation can improve the understanding of the story in multimedia environments.
It is important to emphasize that the process of double coding only happens with animations that are closely linked to the content of the verbal text of the story. Those that have only decorative function might distract the children, "by posing a high cognitive load on their working memory" (Takacs and Bus, 2016: 11).
Concepts discussed by Linda Labbo and Melanie Kuhn, in the era of digital storytelling on cd-roms, may be of great importance to characterize multimedia application guidelines as suggested above. Labbo and Kuhn conducted a qualitative research to examine the understanding of stories on considerate and inconsiderate CD-ROMS. The researchers defined considerate CD-ROMS as those that included multimedia features that were congruent with the story. The inconsiderate ones are those that include incongruent multimedia features, that is, purposeless, not essential to the story. Results indicated that only considerate CD-ROMS supported the understanding of the story (Labbo and Kuhn, 2000: 206-207).
Another feature of e-picturebooks is the presence of several textual forms. The text does not appear only in typographic form. Most of the time, there is the option to listen to the narration of the story, besides a very common feature that modifies the color of the text in parallel with the narration (text highlighting).
Ofra Korat and Tal Or compared a commercial e-picturebook with an educational one, which featured, in addition to multimedia elements, definitions of difficult words, text highlighting and repetition of words when they were clicked (with particular attention to pronunciation). The researchers call these features "scaffolding mechanisms" (Korat and Or, 2010: 149). From the analysis of the joint reading of 48 pairs of mothers and their children between 5 and 6 years of age, it was observed that the educational e-picturebook led to a more expanding talk on the part of the mothers. Regarding the notion of expanding talk, Korat and Or define the factor through nine levels of conversation content, ranging from concrete information immediately available in the story to higher cognitive processes or abstractions (144). Korat and Or believe that the features of the educational application have increased children involvement in reading activity and "might have directed the mothers to expand the discussion about the story understanding, including [for example] word meanings" provided by one of the features (149).
In the same research, Korat and Or also compared each e-picturebook with its printed equivalent. The results indicated that reading the printed book generated even more expanding talks than digital. In contrast, the reading of the digital book generated more discourses initiated by children and more responses to maternal initiatives compared to the printed book. This may have happened because the expanding support in the e-book is embedded in the program and that may be the reason why mothers have not made effort to expand the text by themselves. "The expansion which is built into the e-book program is similar to the adult's expansions during joint printed book reading and is an extra for them" (Korat and Or, 2010: 149).
Also, very common in e-picturebooks, narration can be activated at the beginning of the story, most often by choosing whether we prefer it to be told by the digital narrator or read by ourselves. Specifically, some applications have been tested to help children with reading disabilities. Listening to the narration of a text can help children connect to written words, this way "the hardest part of the fluency is being taken away, so they are able to focus on understanding the text" (qtd. in Guernsey and Levine, 2016: 42). However, it should be avoided that children always use the feature of Read to Me in all readings instead of Read by Myself to practice their reading ability.
In the context of the cd-roms, the researcher Adriana Bus conducted an experiment with 4-5-year-old children in reading sessions of P.B. Bear's Birthday Party, by Lee Davis. That artifact presented challenges and puzzles in one of its reading options, as well as most of the today's e-picturebooks. Bus observed that the children's engagement with the artifact was greater compared to the reading option that did not present the challenges. However, most children were ignoring the narrated text and focusing only on the challenges (DeJong and Bus, 2003). Children had to struggle to understand the story, which demonstrated that unrestricted access to plays in the context of digital narrative could lead to more difficulty in understanding it.
The development of such interactive features and gamification is a trend that has been explored for some time, since when the computer began to be increasingly inserted in everyday life. However, "young children's access to these was limited by [...] the fine motor skills and eye-hand coordination required to manipulate a keyboard and mouse"; capacities not yet fully developed in early childhood (Lovato and Waxman, 2016: 2). Despite that, another research which restricted computer interaction to a single key found different results. That research focused on the transfer of learning, which can be defined as the ability to use information learned from a given medium to reason through events in the real world. In the experiment conducted by Alexis Lauricella et al., they showed three stuffed animals to children, and those toys were hidden in a room later. Three groups underwent different experiments to obtain the information about the place where each toy was hidden. One group watched the events taking place in the real room; another group played with a computer interface that allowed to reveal the location of each toy through a simple interaction; and the last group just watched passively the same interface running. The children who interacted with the computer interface were as successful in testing as those who watched the hiding process in person. The children who just watched the interface running, without interacting, had inferior results. This result suggests that children satisfactorily learn from "specific contingent interactions" (Lauricella et al., 2010: 368).
With the advent of touchscreen technology in mobile devices, the barriers of motor development have been drastically reduced because "most children can become adept at touching, swiping and pinching on the screen" (Lovato and Waxman, 2016: 2). Changes in reading devices and their formats have made it urgent to re-evaluate previous studies and search for positive possibilities in the application of current resources. Researchers from fields of developmental psychology and learning sciences have been analyzing the conditions through which children learn through screen-sensitive interactions, questioning the impact on cognitive and social development.
One of the human cognitive capacities that work to keep the reader following a story is the working memory, which is admittedly limited (Sweller, 2005: 26). E-picturebooks that do not take that into account are bound to fail. "Constant switching between two different tasks, understanding the story on the one hand and exploring games and hotspots on the other, might place too much extraneous load on the working memory" (Takacs, Swart and Bus, 2015: 701). Interactive elements that do not support the story can result in superfluous processing and cognitive overload, which may interrupt the processing of what is essential for understanding the story. Lisa Guernsey and Michael Levine argue that the placement of interactive elements such as plays and puzzles may be a determining factor for the educational value of a digital book. The authors cite the guidance of Alice Wilder as an example to achieve positive results. With respect to location, Wilder emphasizes that there is a great difference between interactivity anywhere and what she calls "interactivity on the plot line" (qtd. in Guernsey and Levine, 2016: 42). It specifically refers to the application of interactive elements closely related to the stories.
Marie-Laure Ryan, a researcher dedicated to interactive narratives, argues that challenges in video games are based on the desire to achieve the stated goal. Therefore, in the context of e-picturebooks, the action of adding interactions as challenges and play should also inherit a narrative goal, which could motivate the interactions of the individual with those experiences. One of the examples cited by Ryan is to present the goal as a mystery that must be solved for the narrative to continue (Ryan, 2015: 242-244).
The interactivity in e-picturebooks is quite restricted, most of the time. All possible interactions and their effects are previously defined in the software. This restriction serves the main purpose of the artifact: storytelling. So, limiting the possibilities of interaction has been a necessity both to avoid negative effects in the process of understanding stories and to reach the narrative outcomes expected by the creator of the story.
Fatma Al Aamri, Stefan Greuter and Steffen Walz argue that there are still few studies that specifically assess the capacity of interactive features to support story comprehension. Researchers emphasize that most of these studies compare printed books to interactive digital books overall, but "different types of interactivity in different e-books could have different effects on comprehension" (AlAamri, Greuter, and Walz, 2015: 7). Lovato and Waxman emphasize the importance of investigating not only the ability to interact with the screen, "but also their engagement with unique features of modern touchscreen devices such as localized content, cameras, and speech recognition" (Lovato and Waxman, 2016: 4).
Regarding the difference in content organization between picturebooks and e-picturebooks, it was not possible to find satisfactory conclusions, or even partial results, simply because of the lack of studies addressing this problem. We emphasize here the need for research into the layout of those narrative artifacts, which may further contribute to the guides of good practices for designers producing e-picturebooks.
In 2015, Zsofia Takacs, Adriana Bus and Elise Swart conducted a meta-analysis of empirical research about digital storytelling. Those authors investigated the effects of stories produced with advanced technologies on the development of children's learning, comparing them to more traditional forms such as printed books. The analysis corpus comprised 43 surveys, involving a total of 2,147 children (Takacs, Swart, and Bus, 2015). Takacs, Swart, and Bus have found a small but significant positive additional effect of the more advanced technologies regarding the indexes of story understanding. The results "reflect the additional effect of technology on top of the benefits of more traditional story presentations", providing evidence that "technology can enhance the effects of storybooks on young children's literacy development" (727).
Zsofia Takacs and her fellows noted that multimedia features were beneficial additions to the stories, with small to moderate positive effects. That result supports the hypothesis that "extra nonverbal information such as animated visualizations, background sounds, and music, as long as congruent with the narration, aid children's comprehension" (Takacs, Swart, and Bus, 2015: 728). In contrast, the meta-analysis showed that interactive elements had no positive contribution to the comprehension of the story since interactivity interrupted narrative linearity (729). This result confirms that interactive features are distracting at some level, and may result in cognitive overload. However, the more closely related to the story is the interactive function, the lower the cognitive cost in the change between the different tasks performed by the readers.
The production of e-picturebooks is still quite recent. We hope that the practices of designers and researchers in various fields capable of cooperating with the development of these digital storytelling experiences can come closer and closer to the ideal interactivity. Overall, however, those experiments with advanced technologies were more successful on story understanding compared to printed books.

IV.CONCLUSIONS
The purpose of this article was to introduce a set of topics about a digital equivalent of the picturebook, reframing them by new theoretical perspectives in situated cognitive science. The argument is that, based on its specific features, epicturebook restructures the way in which stories are conceived and understood; and it may involve different cognitive systems operating in reading and understanding tasks in relation to the printed book.
From the descriptions of characteristics of the e-picturebooks, it was possible to note that, despite their similarities with picturebooks, they are very different artifacts. As researchers have observed in teaching practices that included the use of the iPad, print and digital media have different affordances, corresponding to different modes of writing and reading. Those modes of writing and reading must consequently demand different skills in the process of reading or navigation (Hutchison, Beschorner, and Schmidt-Crawford, 2012: 16).
We show some results regarding understanding of stories that were found in research about each e-picturebook feature separately. Besides that, general data on the effects of stories produced for advanced technologies compared to more traditional forms proved that digital storytelling experiences were more successful regarding story understanding indexes (Takacs, Swart, and Bus, 2015). According to the considerations on distributed cognition made in this article, the positive result obtained in that research does not mean an expansion of the capacity for story understanding made possible by new technologies, but a restructuring in the set of functional abilities in cognitive systems different from those used in reading of the printed picturebook.
The e-picturebook can be understood as a cognitive artifact where the fundamental boundaries of the picturebook are expanded by software programming, which includes new creative possibilities that constrain the ways of dealing with the storytelling issue, while also introducing new issues to the task of understanding illustrated stories in this medium. We conclude this article by emphasizing the argument that the e-picturebook is a distinct storytelling experience from the picturebook not only in its constitutive features but in regard to the cognitive systems involved in the understanding of stories. Thus, we intend to avoid a shallow comparative-qualitative judgment with the picturebook, prioritizing the necessity of the analysis of e-picturebook in its own terms, paying attention to its specific potentials and seeking to develop them through further research.