Neural networks in modern animation art: aesthetic innovations and new horizons

Zaitcev Aleksei Yakovlevich ORCID: 0000-0002-1118-1877 PhD in Art History Associate Professor; Department of Animation and Computer Graphics; All-Russian State University of Cinematography named after S.A. Gerasimov 3 Wilhelma Peak St., Moscow, 129226, Russia art-mary@mail.ru Other publications by this author

The use of neural network tools in animation creativity opens up new aesthetic possibilities and raises important questions about the artistic expression of an animated film. This study focuses on the significant impact of modern technologies on the animation industry and raises the question of the need to adapt professional skills and educational methods in accordance with these changes.

Creating animated video content is a time-consuming process that involves various experts in the field. With the advent of computer technology, many professions are dying out and are now rare. For example, the profession of a graphic artist and a phaser. In their work, modern 2D animators use the professional Toon Boom Harmony tool, which is suitable for cross-frame, frame-by-frame animation and their combinations, creating special effects. Thanks to this program, which allows you to work with various animation styles, it became possible not to draw (and draw, phase) a character, but to move a puppet, getting a ready-made animated image, as, for example, in the animated series "Rick and Morty" (2013-), produced as part of the Adult Swim block on the Cartoon Network TV channel. and "Smeshariki" (2020-) from the Petersburg studio.

The technologies used in cinema are evolving, and Motion Capture (MoCap) technology is emerging, which can be used to create character animations without manual labor. MoCap allows you to capture the movements of actors using special cameras and markers that are fixed on the body. After recording the movements, the data is processed by software, which converts them into digital animation. The advantages of MoCap are not only to speed up the animation creation process, but also to promote a high degree of realism of movements, as well as the ability to capture complex movements, including facial expressions. Instead of manually applying animations to 3D models, MoCap allows you to capture movements, which reduces development time and increases efficiency.

Technological progress does not stand still, and currently the capabilities of neural networks that create visual content in image and video format are actively used in modern film production. A neural network (hereinafter referred to as a "neural network") is a mathematical model, as well as its software or hardware implementation, built on the principle of organizing biological neural networks — networks of nerve cells in a living organism. The concepts of "neural network" and "artificial intelligence" should be separated:

Artificial intelligence (AI) – "ĸомплеĸс technological solutions that allow to simulate ĸогнитивные фунĸции человеĸа (вĸлючая self-learning and поисĸ solutions without pre-specified algorithm) and get when you run ĸонĸретных task results, comparable, ĸаĸ low, the results of intellectual activity человеĸа" (from the Decree of the President of the Russian Federation of 10 оĸтября 2019 No. 490 "On the development of artificial intelligence in the Russian Federation", which refers to the formation of a National strategy for the development of artificial intelligence for the period up to 2030).

A neural network is "a method in artificial intelligence and a program that works with both information processing and a large amount of data, created by analogy with the human brain, which motivates our contemporaries to study innovative inventions" ^[1].

Some authors believe that there are three areas of AI, "in which it is considered as a theory or field of research.": 1) emphasizing the similarity of AI to human intelligence based on the results of actions;2) the emphasis on the fact that AI programs are capable of learning and self-development (neural networks, deep learning, etc.); 3) the definition of AI as a software and hardware complex, more or less capable of solving problems and making decisions (more often to assist in their adoption), as well as mastering other intellectual functions" ^{[2, pp.17-18]}. Currently, AI is increasingly being used in the animation industry, where all three of the above areas are clearly visible: "it is at the stage of active growth and development, demonstrating impressive results in areas such as automating character animation, generating realistic backgrounds and environments, as well as optimizing rendering and post-processing processes" ^{[3, pp. 23-33]}.

The active development of neural network software products allows you to create visually appealing video content in a short time. Among the popular neural network tools are the following:

- the Synthesia platform (URL: https://neural-networked.ru/synthesia / (accessed 10/23/2024), which turns text into realistic videos with virtual presenters;

- neural network Pictory.ai (URL: https://pictory.ai/?el=2000b&htrafficsource=pictoryblog (accessed 10/23/2024), specializing in converting long texts into short video clips;

- an online video creation tool InVideo (URL: https://invideo.io/studio (accessed 10/23/2024), offering a variety of templates and ease of use;

- a tool for creating videos based on text scripts using Deepbrain's "artificial intelligence" (URL: https://www.yeschat.ai/ru/t/deepbrain-ai (accessed 24.10.2024);

- Runway ML Gen-2 neural network (URL: https://runwayml.com/research/gen-2 (accessed 24.10.2024), which generates realistic scenes with smooth transitions between frames based on a text description or image;

- the Russian-language neural network Kandinsky Video (URL: https://www.sberbank.com/promo/kandinsky / (accessed 24.10.2024), which creates short videos and animations based on text queries;

- Genmo video generator (URL: https://www.genmo.ai / (accessed 24.10.2024), which allows you to create videos based on text queries or images;

- Leonardo AI neural network (URL: https://leonardo.ai / (accessed 24.10.2024), which is a powerful platform for image and video generation using advanced artificial intelligence technologies.

These and many other video content generation platforms offer a variety of functions, from turning text into videos with virtual presenters to generating complex scenes based on text descriptions or images, which opens up new horizons of creativity.

Figure 1. Leonardo AI neural network. Screenshot from open sources. URL: https://app.leonardo.ai/auth/login?callbackUrl=%2F (accessed 11/29/2024).

At first glance, the above tools allow users to create videos without the necessary knowledge in the field of animation laws and principles of video editing. As before, when computer technology came to animation, many directors and artists were fascinated by the possibilities it offered. Technology became more and more accessible, and an ordinary student could create animations on his home computer. Gradually, it became clear that without professional support of the creative process, it is impossible to create a decent, high-quality animated work in all respects. On the one hand, AI technologies make animation creation accessible to a wide audience, on the other hand, it also creates the risk of a large amount of low-quality content. In addition, it must be recognized that "excessive interest in AI can lead to the loss of a unique author's style and creativity in animation" ^[3]. Anyway, for the full-fledged, professional use of neural network technologies in cinematography, "it is necessary to have qualified personnel, high-performance equipment and high-quality Internet access" ^{[4, p. 41]}.

Using the capabilities of neural networks to create animations opens up new horizons in this area, greatly simplifying and speeding up the production process. The aspects of using neural networks in the animation industry are extensive, and this is primarily motion generation. Neural networks can recreate realistic movements of characters, which in the future may well replace the expensive motion Capture technology, which is widely used in films, animation, and video games. Neural networks allow you to create character movement with a high degree of confidence. The synergy between Motion Capture and neural networks promises to revolutionize the approach to animation, making it more efficient and realistic. These technologies allow you to create content with a high degree of detail and emotional expressiveness, opening up new opportunities for creativity in movies and video games.

As an example of the synthesis of Motion Capture and neural networks, we can cite the animated film "Visionary" (VGIK, 2022) directed by Vsevolod Bulavkin, who "... chose [the story of] Bosch because he [Bosch] is a denouncer of sins in everyday life, and the author "wanted to talk about the inner nature of evil," Bosch, according to director, was his "production designer, director, cameraman" (URL: https://ya.ru/video/preview/13033092693491446068 (accessed 23.10.2024).

Figure 2. A frame from the movie "Visionary". Screenshot from open sources. "The film "Visionary" tells about the life and work of the great artist Hieronymus Bosch. [The author] tried to convey the complex process of creating a picturesque painting, the sharpness of the artist's feelings and perceptions, and his immersion into his own fantastic inner world." (URL: https://vgik.info/today/creativelife/detail.php?ID=11905&ysclid=m443mbwe3u223383304 (accessed 23.10.2024).

Figure 3. A frame from the movie "Visionary". Screenshot from open sources.

Another example of the use of neural network technologies is present in the animated film "The Ant and the Dragonfly" (VGIK, 2024), directed by Artemy Zaitsev, where backgrounds (decorations) for scenes were created using this tool. The film is eclectic in terms of the choice of technologies: there are 3D (characters generated and animated in the Blender program), three-dimensional layouts made by hand (panorama with the Ant house), and backgrounds made using the Leonardo AI neural network, as well as effects such as rain, snow, falling foliage, made in the program. Adobe After effects, and finally, Adobe Photochop, which is already familiar to everyone. The search for character images was also carried out using neural network tools. The story told in the film is an alternative to I.A. Krylov's fable and tells about what would have happened if the Dragonfly had not come to the Ant in winter.

Figure 4. Screenshot of the workflow for creating the scene of the movie "The Ant and the Dragonfly". The process of creating a scene where the character is made in 3D, and the background is generated by neural networks based on effects. The material from the archive of the film director is provided for this article.

Figure 5. Screenshot of the workflow for creating the scene of the movie "The Ant and the Dragonfly". The process of creating a scene where the character is made in 3D, and the background is generated by neural networks based on effects. The material from the archive of the film director is provided for this article.

Figure 6. Screenshot of the workflow for creating the scene of the movie "The Ant and the Dragonfly". The process of creating a scene where the character is made in 3D, and the background is generated by neural networks based on effects. The material from the archive of the film director is provided for this article.

In the context of automated video content creation, it is important to develop clear criteria for evaluating its quality. The main criteria can be aesthetic appeal, harmony of composition, originality, emotional expressiveness and accuracy of image generation. These parameters will help determine to what extent the created video content meets the high standards of the film industry.

One of the most important aspects of neural network capabilities is the improvement of graphic images: the technology allows you to "stretch" the quality of graphics and visual effects, removing noise, and increasing image resolution, which contributes to the attractiveness of animation, while "the quality of AI-generated backgrounds often surpasses the results of artists' handiwork, providing an unprecedented level of realism and immersion of the viewer in the animation world."" ^[3]. And, in general, if you pay attention to the image quality, then the limitation here is, sometimes, only the user's access to the "resolution" of image generation. "Image resolution", in this case, is a key parameter that determines the quality and detail of digital images and is measured in the number of pixels per unit area, most often in inches, and is designated as DPI (dots per inch) or PPI (pixels per inch), depending on the context. Professional access, of course, will allow higher-quality visual content.

A distinctive feature of neural networks is their ability to learn. For example, while working on the animated film Spider-Man: Across the Universes (2018), "... a team of special effects creators recreated all the drawings and poses created by the artists in 3D and using an artificial intelligence program tried to teach her how to guess what the next frame would be, correcting her mistakes each time. So they gradually taught her how to achieve the result that the creators wanted..." ("How Spider-Man: Across the Universes was invented, drawn and animated." URL: https://vk.com/@pixel_ae-kak-pridumali-narisovali-i-animirovali-cheloveka-pauka-chere (accessed 23.10.24). In addition, artificial intelligence technologies were used to outline the frames in order to give the film the desired style.

Figure 8. A frame from the movie "Spider-Man: Across the Universes". Screenshot from open sources.

The fundamental aesthetic issues in the use of neural networks in cinematography, especially in modern animation art, include a number of key aspects that relate to both artistic expression and audience perception. Firstly, it is the authenticity and originality of the visual series or image. Neural networks, as a rule, are trained on large amounts of existing content. This leads to the fact that the works they create can be perceived as an imitation or compilation of existing styles and plots (if this was not the user's goal). The relevant question is how original the works created by AI are and where the boundary line between plagiarism and authorship lies. Secondly, it is emotional expressiveness: neural networks create visually appealing images, but you can often see that these images lack depth in conveying emotions. Characters can look lifeless and static, such images are suitable for the designation "cold", and this negatively affects the perception of the image by the viewer.

In addition, the modern viewer already has some "viewing experience" that allows him to identify images created by "artificial intelligence", that is, to guess the "handwriting of AI". This is indicated by the results of a user survey at the end of 2023 and in October 2024. We interviewed 25 people between the ages of 17 and 70 to determine who created the image: a human or a neural network. The subjects were offered 10 images, 5 of which were created by contemporary artists without the use of neural networks and 5 using the Leonardo AI platform. In 2023, the subjects identified 72.8% of the images created by neural networks; in 2024, this figure increased to 88%. At the same time, images created by humans were identified as such by 86.4% in 2023 and 82.4% in 2024. It can be assumed that the viewer's ability to identify "digital" images is increasing. Interestingly, images identified as man-made decreased by 4%, meaning subjects in 2024 were more likely to mistake for "digital" images created by artists, which may be related to the subjects' expectations of seeing "digital" content. Let's visualize this data in a diagram.:

. 4pt; text-align: justify;">

Figure 9. The subjects' definition of images created by artists manually and using neural networks (using text editing) in 2023-2024.

This study was an experiment that cannot yet guarantee a high degree of reliability of the results obtained, but, at the same time, is of interest in studying the viewer's perception of images generated by AI and needs further study, which goes beyond the scope of the article.

The massive popularization of neural network tools contributes to the fact that now everyone can create "their own unique image" using artificial intelligence, for example, in the Masterpiece application (a neural network developed on the basis of YandexART, which allows users to generate images, short texts and videos). The user's imagination is limited only by his experience in formulating queries to the neural network (promts). At the same time, there is always a surprise effect when using such a tool: during the entire time of image generation, the user can only guess what he will see as a result. Due to the unpredictability of the result, it becomes more difficult to work on projects when it comes to animation. Nevertheless, it should be emphasized that the use of neural network tools allows a person who is far from creativity to develop creative skills. The stimulation of creativity helps to increase intellectual potential. It is known that creativity activates cognitive processes and forms a personality, which in turn contributes to a more complete disclosure of intellectual abilities.

Animation uses stylization and hyperbolization to enhance the expressiveness of characters and their actions. Neural networks may have difficulty creating such accents without losing realism, which ultimately leads to the need for human intervention to achieve the desired artistic and aesthetic effect.

A characteristic feature of neural networks are problems with the accuracy of image generation. For example, anatomical errors and artifacts that occur during image generation are very common. For example, a human may have three arms or eight fingers on one hand, while a rhinoceros may have two or more horns or five legs. This casts doubt on the idea of fully automating the creative process.

Figure 10. Image generated by the Leonardo AI neural network. An example of "artifacts" or anomalies of images created using neural networks. The specified promt: "an animated character, a girl and a goose." As you can see in the image, artificial intelligence has created a hybrid of a girl and a goose from the character on the right. In addition, for some unknown reason, the neural network placed the central character waist-deep in water.

Nevertheless, neural networks can create unique visual styles, generate images and animations from photorealism to abstraction, which allows artists to experiment with new visual languages and images and create unique works. An important task for the user is to learn how to formulate tasks (promts) as accurately as possible.

In addition, neural networks contribute to the acceleration and optimization of the production process.: Artists can now upload sketches, and the neural network converts them into detailed and high-quality images. For example, the creators of the CGI series "Eve. Communication through Time" (2023) used text-to-image neural networks to produce concept art. As the creators of the project note, "neural networks accelerated the work on the script by two times: the invoice was found and checked quickly, experts answered specific questions, which was also faster. On a large project, such optimization can speed up the process significantly, and no, there is no question of replacing living people with machines, it just becomes easier and more enjoyable to work" ^[5].

Figure 11. A frame from the movie "Eve. Communication through time." Screenshot from open sources.

With the help of a neural network, a screenwriter can quickly visualize ideas and concepts, just enter the necessary test scripts. Neural networks can automate the process of creating scripts, characters, and dialogues for animated films, which significantly reduces development time and increases the originality of works. For example, during the work on one of the episodes of the animated series Prostokvashino (Soyuzmultfilm Film Studio), the Kandinsky neural network was used, among other things, to write a script, search for references and artistic solutions. And in one episode, a supporting character appeared – a chicken, which was generated by artificial intelligence ^[6].

It is appropriate to touch upon a number of ethical issues that may arise in the process of using artificial intelligence technologies in artistic creation. There are concerns that the use of AI to create animation may devalue the work of artists, as the rapid high-quality generation of animation content by a neural network may reduce the demand for animation artists and related professions. So far, these fears are groundless, since the quality of the product produced by neural networks still requires man-made intervention and, as S.L. Urazova rightly notes, "products involving AI are still far from creativity and the formulation of ideas aimed at the mental development of the individual, his ability to think reflexively. This can be seen primarily in audiovisual products, in music videos, and in the creation of data according to familiar patterns, where artificiality prevails in images, there are signs of experimentation, far from professionalism and a creative approach" ^{[1, p. 144]}.

It should be especially noted that the use of neural networks in animation (and not only in animation) raises copyright issues. Who will be the author of the work: a neural network or a human? In most cases, copyright traditionally belongs to the person or group of people who directly created the work. But in the case of neural networks, the question of who is the author is open. Neural networks are often trained on copyrighted materials, and therefore there is a risk of violating the rights of third parties when using the generated content for commercial purposes. Some authors believe that the rights may belong to the user who initiated the creation of the content, while others argue that the developer of the neural network may also claim copyright. This is a very interesting and relevant question, but there is no exact answer to it yet. Developing the idea of copyright, can we assume that the "author's" set of words, descriptions, and tasks that underlie the creation of an image by a neural network automatically makes this image "copyrighted"? If we turn to the algorithm of image creation by neural networks, we will find that any image they generate is a "collective", compiled image and has a certain number of images underlying it. The finished images are "disassembled" by neural networks into pixels and then assembled into a new image. There is a problem of copyrights to the results of neural networks. In user agreements, some companies write that the result of the neural network is the intellectual property of the company ^[7]. Thus, not only the fundamental aesthetic issues of using neural networks in animation art, but also the issue of copyright, require in-depth analysis and discussion.

Neural network tools provide truly limitless opportunities for an artist, and "talking about whether neural networks have limitations is like answering the question of whether the Internet has limitations. Neural networks are a technology, and how the person who works with it uses it depends on their imagination and ingenuity" ^[8].

Summing up the work, we will highlight its key points.:

Modern technologies such as Toon Boom Harmony software and Motion Capture (MoCap) system have significantly changed the animation creation process. These tools have automated many time-consuming steps, resulting in shorter production times and more realistic animation work.

Currently, neural network technologies are being actively integrated into animation production, opening up new opportunities for motion generation, creating backgrounds and special effects, writing scripts, etc. These technologies simplify the content creation process and make it accessible to a wider audience, although they create the risk of a large amount of low-quality content. Thus, despite the availability of technology, the creation of high-quality animated content requires professional support, which emphasizes the need for qualified personnel and high-performance equipment to fully exploit new technological opportunities.

Using a combination of MoCap and neural networks can revolutionize animation production, allowing you to create more realistic and detailed content. An example of successful integration of these technologies is the animated film "Visionary". Of particular interest is the synergy of neural network tools with other technologies: traditional (layouts) and computer graphics (3D, 2D): an example is the animated film "The Ant and the Dragonfly".

Neural networks, although they can improve the quality of graphics and special effects, raise questions about preserving the originality and authenticity of animated works. This is important to ensure uniqueness and artistic expression in animation. The emergence of new technologies also requires a revision of the criteria for evaluating the quality of animation created with their help. Standards are needed that take into account both technical and artistic aspects. In addition, it is important to consider the long-term effects of AI and neural networks on the author's style and creativity in animation.

The use of artificial intelligence technologies in animation opens up new opportunities for creativity and simplifies the process of creating video content. However, it also poses a number of serious ethical, legal, and professional issues to society. An active discussion is needed to develop approaches to solving these problems, and it should be emphasized that "regulation of the use of artificial intelligence in audiovisual art is also necessary in the legislative sphere" ^{[4, p. 44]}, otherwise video content may be overflowing with low-quality materials.

Despite technological advances, it is important to keep in mind the human aspect of creativity — emotional depth and originality that are difficult to reproduce using algorithms. Combining technology with human art can lead to the creation of unique works, but only if a balance is found between automation and creative control.