"Using Metaverse technologies to conduct virtual multi-user excursions: The experience of the Borodino Battle Panorama Museum"

Akhtamzian Nurlan Ildarovich ORCID: 0009-0002-2302-7778 senior lecturer; Faculty of History; State Academic University of Humanities specialist; ITMUS.ART Studio specialist in exposition and exhibition activities; 'Ostankino and Kuskovo State Palace and Park Museum-Reserve' 28/1 Dekabristov str., Moscow, 127273, Russia, 225 sq. nur@itmus.art Ahtamzyan Amir Ildarovich ORCID: 0000-0002-8367-9156 Director; ANO SIC 'ITMUS' 127273, Russia, Moscow, Dekabristov str., 28k1, sq. 225 info@itmus.ru

Nosova Ksenia Aleksandrovna Senior Researcher; Multimedia Programs Sector ; GBUK Panorama Museum 'Battle of Borodino' 38 Kutuzovsky Ave., Moscow, 121170, Russia, from 1 adm1812panorama@gmail.com

Introduction

Modern technologies have a significant impact on the museum business, opening up new opportunities for presenting cultural heritage and interacting with visitors. One of these innovations is the use of metaverse technologies to create virtual multi-user excursions. Virtual tours expand access to museum exhibits, overcoming geographical and physical limitations, and attract new audiences.

The purpose of this article is to review the experience of creating a virtual multi-user tour based on the Borodino Battle Panorama Museum using metaverse technologies.

The objectives of the article are to explore the possibilities of virtual multi-user excursions for the preservation and presentation of cultural and historical heritage on the example of the implementation of such an excursion in the Borodino Battle Panorama Museum in 2023-2024; a brief overview of the history of the development of virtual exhibitions - from web pages to metavers; description of the process of 3D digitization of museum exhibits and their integration into the virtual scene; analysis of the Spatial platform as a tool for conducting virtual excursions; assessment of the potential of metaverse excursions to increase the accessibility of museum facilities to a wide audience, including people with limited mobility and users from remote regions; as well as assessment of the novelty and importance of metaverse technologies for museum practice in Russia.

The methods used in the study include photogrammetry to create digital copies of exhibits, as well as the development of virtual scenography using gaming technologies. The article presents the results of the introduction of virtual excursions into museum practice and discusses the prospects for further development of this area.

Virtual Exhibitions: from Web pages to metaverse

The format of a multi-user virtual tour to Metaverse is a development of ideas for publishing structured collections on the Internet and virtual exhibitions. Since the 1990s, one can observe the evolution of museums' presentation of their digital collections: from simple databases and CD/DVD discs with the first attempts to accompany structured textual information with illustrative photographic materials to virtual exhibitions on the Internet, in our time [1 p. 23-27]. The very concept of "Virtual exhibition" is ambiguous and "based on the context in which it is used, it is obvious that the authors put a variety of meanings into it" [2 pp. 97-101]. Thus, "virtual exhibitions", as well as "virtual tours", are collections posted on the Internet, including the ability to filter by characteristics, containing photo and video materials about the exhibit, and related 360-degree panoramic photographs of the exposition and exhibitions, often accompanied by points of interest with the ability to view individual exhibits with a label. The logical development of these ways of presenting digital collections has become the possibility of presenting them in a three-dimensional, three-dimensional format: "the use of multimedia technologies allows a static image to be made three-dimensional (3D), dynamic (supplementing with a video sequence or animating), accompanied by text, musical accompaniment, verbal comments, that is, to combine various types of information, to influence not only rational, but also the emotional sphere of a person" [3 pp. 207-217].

First of all, this development was facilitated by an increase in the speed of Internet access, an increase in the availability of tools for 3D digitization, having gone from expensive 3D scanners based on structured illumination to mass tools for creating 3D copies of museum objects based on multi-angle photographs (photogrammetry).

In the early 2020s, Metavers became one of the most popular concepts for the further development of the Internet. The term "metaverse" was first used by American science fiction writer Neal Stephenson in his science fiction novel "Avalanche" ("Snow Crash"), published in 1992. In this novel, Stevenson described a virtual world into which people can immerse themselves using avatars, interacting with each other in three-dimensional space. In the general virtual space, users are represented and identified with "avatars" - three-dimensional bodies of users. Interest from the largest social networks and brands has led to the creation of virtual spaces for collaboration, representative offices of manufacturers of clothing, food, cars and appliances, as well as galleries of digital artists and designers as places of sale of "digital originals" using blockchain technology.

To date, platforms such as Roblox, Fortnite, Minecraft, Decentraland, VRChat, Sandbox, and Spatial meet the criteria of "metavers" most closely. Most of them can be visited through an application or browser on a computer, a mobile application or a VR headset. Many of these platforms were originally gaming platforms. Some of them support the ability to import their own virtual scenes, and some provide tools for creating scenes by the mechanics of the platform itself. At the end of 2024, public interest in the world in the field of technological progress, as a whole, switched to the development of AI, and the meta-excitement decreased, steadily occupying certain niches of human activity, certain conclusions can be drawn. Any created virtual space needs a carefully planned set design: gaming platforms were able to keep the audience thanks to activities inside the stage: competitive, focusing on social interaction inside the stage, including tools to influence the world in the scene and the appearance of their avatar (the character on whose behalf they interact with the stage and other participants), and also through the creation of one-time thematic events. Many metaverse spaces disappeared shortly after their appearance due to the fact that visitors had nothing to do in them.

The museum exposition is a space that attracts the attention of visitors and evokes emotions. Just like in the original exhibition, the metaverse exhibition is a channel of communication with the virtual visitor. The concept of a "digital twin", which appeared in 2010 at NASA, implied the creation of the most accurate virtual representation of a real physical product, gradually penetrated into various areas of human activity: engineering, construction, energy, urban planning, etc. areas [4 pp. 35-43]. The digital double of a museum object can in fact be a container, and include the most detailed information about it: a highly polygonal three-dimensional model, with the transmission of verified optical and color features in the form of a set of textures, metric characteristics, exhaustive textual characteristics, gigapixel photography, an archive of multi-angle photographs taken not only in the visible range, but also in infrared, ultraviolet and other ranges, as well as include sound and video materials, including in different temporary states of the object - during the period of existence, before and after restorations, in different states of preservation, as part of a complex of objects. With the help of animation, technical exhibits can be shown at work, natural science exhibits can be shown during life. Unlike other applications, such as building BIM models of buildings, the inclusion of such detailed information about the exhibit is still far from practical implementation.

Conducting excursions in the metaverse space of the museum can become another channel of interaction with the audience, and the experience that a visitor can get on the one hand resembles interaction with social networks, thematic video conferences, and on the other hand is a simulation of a real visit to the museum space, somewhat resembling a computer game. A number of cultural institutions have begun to try metaverses to present their exhibits: the Tate Museum in Minecraft ^[5], the Metropolitan Museum of Art ^[6] and the Computer History Museum in Roblox ^[7]. In Russia, regular metaverse tours are conducted by the Sergei Kuryokhin Center for Contemporary Art, which has copied both the building itself and the interiors of the halls with exhibits in 3D ^[8].

Digital copies of museum objects presented in the metaverse scene in the context of the exposition, as well as physical originals, have properties that are defined as basic in museology: informativeness, expressiveness, attractiveness, representativeness and associativity. The display of digital doubles of museum objects in the metaverse scene, as well as works of art originally created in digital form, has a number of advantages: they can be visited without queues from anywhere in the world, including around the clock in a free inspection scenario, increasing accessibility for both a remote audience by geographical location and for visitors with limited mobility. Thematic group metaverse excursions can become educational material in schools and universities. For the museum, this is an opportunity to create an archive of exhibitions, exhibit dilapidated and externally sensitive exhibits that have been lost or virtually reconstructed, as well as to test the hypotheses of the exhibition's design before its actual construction. The visitor can examine the subject comprehensively, and as part of the tour group ask a question to the guide and discuss with the participants. Summarizing all of the above, "thanks to the latest technologies, art is experiencing a "digital rebirth": various ways of interacting with viewers and collectors appear, geographical and social boundaries are erased" [9 pp. 203-212].

When developing a museum metaverse excursion, it is worth considering that the existing metaverse technologies, both in terms of the technical characteristics of viewing devices and in terms of the digital content of the scenes themselves, are at an early stage of development. It is important to note that metaverse scenes can be visited from various devices: computers, phones, game consoles, VR headsets - all of them have different technical characteristics, so launching on some of them is completely impossible or difficult. A physical visit to a museum is often accompanied by a unique sense of presence and interaction that is difficult to fully reproduce in a virtual format, and existing most immersive devices such as VR headsets are still far from providing an equivalent experience. There are a number of objective reasons why VR devices, having occupied a certain niche, have not become truly mass devices: wearable headsets still have quite large dimensions, a small battery for autonomous operation, insufficient screen quality and performance to launch photorealistic scenes, some models require connection to a computer, in addition, a small percentage For users, moving around the virtual scene causes the so-called "Marine cyber pain" ^[10]. The scenes placed in the metaverse often differ in primitive graphics - in order for them to load faster, they have to be compressed very much, and not all headsets have the ability to use foveal rendering, in which only the currently observed viewing angle is calculated in high resolution. The capabilities of the servers responsible for storing and transferring scenes to the user's device are also limited, as well as multiplayer functions that allow the simultaneous presence and communication of many participants.

Metaverse tour "1812: Images of the Epoch"

Work on the three-dimensional digitization of museum objects in the Borodino Battle Panorama Museum began in 2018. At that time, the main building of the museum was under long-term renovation and three-dimensional models were used as illustrative material in a series of short popular science films produced by the museum to communicate with its audience during that period. In 2020, one of the first Russian museums prepared the exhibition "The Game of Soldiers", which could be viewed in 3D AR mode (augmented reality using a phone) on the platform of the Ministry of Culture Artefact. It consisted of objects digitized in 3D from the museum's collections, and it was possible to get acquainted with them accompanied by scientific texts and an audio guide ^[11]. Since 2019, the museum has been placing digitized models on the Sketchfab platform in the public domain, without the need to install additional software for viewing. In total, since 2019, the museum has hosted about 60 models on this platform with a total number of views of 22 thousand. In 2021, the museum presented a project for the virtual reconstruction of the old Panorama building, built in 1912 on Chistye Prudy and not preserved to our time. The work on the virtual reconstruction, including the internal structure, was carried out on the basis of a layout, photographs, archival drawings and descriptive sources. The results of the work were presented at scientific conferences, including international ones [12 pp. 45-50].

The qualitative development of this direction in 2023 was the development of the panorama museum "The Battle of Borodino", together with the ITMUS studio.ART of the virtual multi-user tour "1812: Images of the epoch". The new format of the interactive virtual multi-user tour was designed to draw the attention of a young audience to the aspects of history studied, preserved and popularized by the museum, using the most modern tools for working with 3D: from photogrammetric digitization to character animation and the use of visual particle effects. The development of the excursion scenario was carried out by a research associate of the museum, and scientific consultation with specialists, historians and researchers, in our opinion, allowed us to approach the issue of popularization of historical and cultural heritage with scientific certainty.

An interesting analogy can be drawn between panoramic art and VR. Panoramic paintings, especially popular in the 19th century, like VR, created the illusion of the viewer's presence inside the space, surrounding it with a 360-degree image. The main purpose of the panorama was to reproduce the real world so skillfully that the audience could believe in the authenticity of what they saw ^{[13, p. 49]}. The audience felt that they were "inside" the stage, and not just looking at it from the outside. It was important to convey the feeling of moving to another place and time by various means: the geometry of space and the location of the observation deck, the most realistic lighting, an attempt to create topographically accurate and realistic reproductions of a real place through a subject plan ^{[13, p. 63]}. Panoramas were the first to discover the idea of creating the illusion of boundless space in an enclosed space. The moving diorama background became an integral part of theatrical productions of the nineteenth century, and it was panoramic painting that influenced this ^{[13, p. 66]}. In addition, sound effects were also used for greater realism.

In this regard, to create a metaverse panorama scene, there was a task of an integrated approach to the issue of digitizing exhibits: creating a 3D copy of the painting, the subject plan, the arrangement of lighting in the scene, the location of the observation deck. To convey the effect, when climbing the stairs you see an array of a reflector above you and gradually get into the three-dimensional space of F.'s painting.Rubo "Borodino" (one of the largest panoramas in the world, the size of the painting is 115 m by 15 m.) the space of the lower hall and the staircase itself was modeled. In addition, this hall, not being an exact copy of the present, contains showcases with digitized museum objects: cold steel and firearms, ammunition, personal belongings and field surgery items, stationary mannequins with images of that era with which you can interact - when you press a button, you can literally try on these images.

The preparatory stage for the implementation of the metaverse excursion at the Borodino Battle Panorama Museum was the preparation of a scientific and technical concept of the project. It described the main tasks to be solved, the target audience, the stages of launching the project, a list of objects for digitization, technical parameters of equipment and software for digitization and stage placement. Attention was paid to risk management with alternative project implementation options, and time was taken into account for testing the tour before launch.

To digitize museum objects, actors in images of that time, as well as the subject plan and parts of the hall, the photogrammetry method was used, a method for obtaining three-dimensional models and an array of different-angle photographs. In total, more than 10 thousand photos were taken during the work on the project.

The subject plan is an important artistic part of the space, smoothly passing into the painting itself. The distance from the viewing platform to the painting is about 13 meters, according to F.Rubo, the viewer's presence on the observation deck should take him to the center of the burned village of Semenovskoye, from where he watches the unfolding picture of the battle at 10:30 a.m. A reflector located above the observation deck specifically diffuses the lighting, directing light at the painting at a certain angle. During the assembly of the scene on the Unity game engine, this effect was transmitted through the creation of a reflector model with an imitation of the fabric surface and the arrangement of light emitters.

To digitize the three-dimensional subject plan, photography was carried out with a SLR digital camera from a tripod, on which a stabilizer with panoramic photography functions was mounted. Moving the camera from place to place, more than 2,500 subject-plan photographs were taken. For the best "gluing" of images at the stage of photogrammetric processing, special circular labels printed in A4 sheet size were pre-laid out throughout the hall, which were removed on the final model. The output model of the subject plan had a size of 5 GB. with a texture quality of 8K. A digitized copy of the painting by F.The Borodino rubo as 32K was kindly provided by the Museum Plus company and combined with the subject plan at the stage of stage assembly. The space of the hall, the observation deck, the staircase leading to it and the lower hall were created using three-dimensional modeling, including recreating the properties of surfaces, for example, the blocking properties of the floor (Fig.1).

Fig.1. The view in the scene of the subject plan, the observation deck and the painting.

When shooting objects such as cold steel and firearms, field surgery items and ammunition, subject shooting with constant studio lighting sources was used. Color and size verification was carried out using a color target and a ruler. To automate the process of shooting individual objects, a motorized turntable with a set offset angle was used. A number of items were filmed using a mobile phone camera and a telescopic selfie stick, which allowed for multi-level images. Some objects, which, due to their size and weight, were difficult to transfer to the photo zone, were photographed at the exhibition: The carriage and barrel of a 3-pound unicorn of 1827 (Fig. 2), as well as two full-size models of horsemen: a sergeant of the 5th Cuirassier Regiment of the Great Army and an Ordinary of the Order Cuirassier Regiment of the Russian army. One of the three-dimensional models of the riders was later animated in specialized software (Fig.3). On average, 300-600 photos were taken to obtain one three-dimensional model of an object. Some items, for example, a telescope, an infantry flintlock rifle of the 1808 model, an infantry soldier's cleaver of the 1807 model. For comprehensive visibility of the object, they were removed from two sides, and the resulting models were glued together at key points, and the 1804 hiking chest was removed in two states - with an open and closed lid. Some objects flashed when shooting, especially those with a lacquered surface, and a polarizing filter was used to cut off glare, which often turns into geometric dips at the stage of model construction.

Fig. 2. The process of photogrammetric photography of a museum object.

Fig.3. Screenshot from the program. Animation of a galloping horse.

Shooting actors in the images of that time required a fundamentally different approach. Museum items are static, so shooting each one can take several hours. With people, such photography is difficult due to their constant mobility, which negatively affects the quality of the final model. Therefore, to photograph the actors, it was decided to use a special photo studio, where filming is carried out to create digital three-dimensional copies of people for movies, advertising and computer games. It is arranged in such a way that the actor enters the center of the set, and from all sides he is surrounded by 150 digital SLR cameras that are synchronized with each other for simultaneous shooting (Fig.4). The actors in the historical images of the era of 1812 were professional reenactors who devoted many years to studying the authenticity of the reconstruction of the images of that era: Ilya Ulyanov, Dmitry Kardogin, Natalia Kolesnikova, Timur and Artem Belov. To simplify the subsequent processing of the model, as well as to ensure that the three-dimensional elements of the body do not "stick together", the actors are shot in a special T-pose, with their arms raised horizontally to shoulder level or in front of them. Before shooting, a matting spray based on talc was used to treat highly reflective surfaces (camping flask, drum elements, etc.). Based on the results of the photography, the studio transferred an archive of photos in the original raw format, which was processed in photogrammetry software, as well as other shooting objects to obtain a three-dimensional model (Fig.5). In total, 8 models were prepared with military and civilian looks of Russians and Frenchmen of the beginning of the XIX century: a company drummer of the Lithuanian Life Guards Regiment, a grenadier of the Semyonovsky Life Guards Regiment, a colonel of the Cavalry Regiment, a sutler, an officer of the Semyonovsky Life Guards Regiment, a grenadier of the 1st regiment of foot grenadiers of the Imperial Guard, an image of a lady in a ball gown of the XIX century, a set of a ballroom uniform of a colonel of the Cavalry regiment. Some of these characters were animated to create an interactive opportunity for visitors to try on these images and move around the stage in them. In addition, these images are used during the tour in a virtual scene: telling about the branches of the armed forces, the guide can literally put these images on himself (Fig.6).

Fig. 4. Avatar photogrammetry studio. The process of digitizing images.

Fig.5. Screenshot from the program. One of the stages of processing avatars.

Fig.6. Using an avatar in the image of the era of 1812 to move around the stage.

Reality Capture and Agisoft Metashape software were used for photogrammetric image processing. A set of multi-angle photographs was loaded into the programs, which went through several stages of processing: image alignment to obtain a sparse point cloud, building a dense point cloud, polygonal model, texturing. Also, the tools of this software allow you to trim unnecessary elements of the environment and details of models, simplify polygonal geometry, and smooth the surface. In some cases, the geometry of the models required manual modification. For these purposes, sculpting tools were used in the 3D Coat and Blender programs. For manual refinement of textures, including the application of layers transmitting the effects of reflection, gloss, surface roughness (roughness, metalness, etc.), Substance Painter software was used. The models of the actors were animated in the programs AccuRIG ActorCore and Mixamo. The AccuRIG ActorCore software allows semi-automatic marking of humanoid models with a skeleton of bones (rigging) and places of folds on the skin (skinning). The marked-up models were animated using the Mixamo online service, which is an extensive library of recorded body movements. The horse animation was implemented using the Rigfy Zoo plugin for Blender, which allows you to semi-automatically mark the horse model with bones and mark the places of bends, as well as animate the movements of the horse in calm and active states. Scene assembly and manual modeling were carried out in the 3D Max, Blender programs and in the Spatial plugin for the Unity game engine. The compression of models and scenes was carried out using the Rapid Compact online service, which allows you to select the compression ratio.

The list of available metaverse platforms was analyzed to place the scene, and the Spatial platform was selected (Fig.7) ^[14]. It provides an opportunity to visit the tour from different types of devices: through a computer browser, without the need to install additional software, through an IOS mobile application or an application for most VR headsets. Through the interface on the website, the museum can upload and display images, presentations, three-dimensional models, videos in most formats in a virtual scene, share the screen, embed links to web pages in the scene. At the same time, there can be 50 participants in the scene, including the guide. There are chat functions, turning on the camera and microphone, and avatars, in addition to the usual movement, can send reactions, gestures, etc. There are requirements for optimizing the size of models and the loaded scene, the scene model itself should weigh no more than 150 MB. and the content is also no more than 150 MB. To do this, a technically difficult task was done to optimize the size of the models of both the scene itself and the loaded objects, each of which, after optimization, began to weigh no more than 8 mb. while maintaining adequate model quality. Special attention was paid to the sound design of the stage, which significantly increases the feeling of immersiveness - the sound of footsteps when moving on various surfaces, as well as the background sound of battle noise, was added.

Fig.7. The process of conducting a metaverse tour "1812: Images of the Epoch" for a group of visitors on the Spatial platform

A website was developed to register visitors, publish information about the project, an encyclopedia of three-dimensional models, as well as detailed instructions for use ^[15]. Upon registration, the visitor can select the date of the next excursion in the calendar and receive an hour before the event a link to enter the tour, and contact where to contact in case of difficulties. The museum staff has formed rules of ethical behavior in the scene, which the visitor must agree with before registering. A knowledge base was created for the project team on the Notion platform, which contains structured information on all models, versions of the stage and site assembly, texts and content for the tour, press releases, and analysis of information on feedback from visitors. To get information about the recordings from the site, a chatbot was developed on the telegram platform, transmitting information to the project team group.

A researcher at the museum, who conducts excursions on an authentic panorama, has developed a 50-minute guided tour program specially adapted to the virtual format.

The technical assistant illustrates her story with three-dimensional models, audio, video materials, digital copies of paintings or fragments thereof. At the end of the tour, visitors are invited to take a small survey for feedback, identify possible problems and collect suggestions for development. In the three tours that have passed since the launch of the project, we have received more than 10 reviews with useful comments.

To popularize the tools that were used to create a virtual stage, an illustrated presentation was posted on the site, which can be used as a methodological guide for creating your own virtual multi-user tour, which lists the necessary software and equipment.

Results and prospects

In Russian museum practice, this is the first experience of practical implementation of a multi-user virtual tour using an integrated approach to 3D digitization.

I would like to emphasize the theoretical significance of the experience gained. It includes the complex digitization of various types of objects, the creation of a virtual scene with thoughtful scenography. The experience of conducting a virtual multi-user tour by a guide differs from conducting an excursion both in the space of a real museum and from a video tour. The guide has the opportunity to present accompanying materials in various forms: in the form of documents, paintings and graphics, audio, video materials, three-dimensional models. The visitor can view them from any angle, degree of magnification, and look at the dynamics. The possibility of interaction between the participants of the scene introduces an element of social interaction and is an important aspect for creating a more interactive and engaging educational environment. Due to this, participants can not only perceive information passively, but also actively interact with the content and with each other, which contributes to a deeper assimilation of the material. This format of excursions allows you to include elements of gamification and contributes to the formation of a collective experience, which distinguishes it from traditional methods of museum work. The specifics of conducting and perceiving such excursions are becoming urgent tasks for further research, especially in the context of analyzing the educational potential of virtual excursions, their impact on the perception of historical material and the development of new forms of museum communication.

In order to attract a new audience and retain the old one, the project can be further developed by increasing the size of the stage with new spaces, digital copies of museum objects, developing various types of excursions, holding one-time thematic events, including the possibility of pre-recording movements, such as concerts or theatrical performances, introducing a game element into the independent study of the scene. To popularize the project, it would be advisable to try out an off-site format with a set of VR headsets to schools and universities, thematic events. Despite the initial stage of development of this form of communication with visitors, it can be concluded that metaverse excursions have great potential to attract a wide audience, including young ones. Also, there is a request to hold on-site VR events in regional educational and cultural institutions for access through such a form to the historical and cultural heritage stored in thematic museums. A potential, but not yet tested in practice, area of work for the development of metaverse excursions is to increase the accessibility of knowledge for people with limited mobility, for whom visiting a real museum often involves a number of difficulties. With increased device performance, increased Internet speed and the advent of new technologies for digitizing the real world, such as NERF, metaverse tours will be able to fully unlock their potential, and museums will be able to use them to popularize their cultural and historical heritage.