EVA Berlin 2018 Conference 7 – 9 November 2018 Kunstgewerbemuseum (Museum of Decorative Arts) Kulturforum Potsdamer Platz, Matthäikirchplatz 8, 10785 Berlin

Spatial Augmented Reality for collective and immersive experiences in museums

GERMAN Ronan, Ph.D., HERVY Benjamin, Ph.D .
Mazedia, France

Very often, given the more individualistic experiences, they allow, digital technologies such as Virtual Reality and Augmented Reality are seen as not conducive for sharing quality time between family members or a school group, (especially when headsets are required!).

Contrary to this belief, our project explores how Spatial Augmented Reality technologies, which will be calling along the article as SAR technologies, can approach group participants with a more socially and engaging physical experience; hence group behavior analysis runs on a SAR device, relying on a 3D triggering animation for facial expression, and gesture recognition.

The experimentation, which will involve 3D animals and provide a live immersive family experience, will take place in the aquarium of Le Croisic, in France


In Spring 2018, Mazedia, a communication agency serving the cultural and tourism sectors, began working with the French-based Croisic aquarium. It is here where the immersive live family adventure using 3D experiments happens.

The project aims to offer aquarium’s visitors with a new experience, giving them a more immersive and engaging experience with the aquarium’s aquatic exemplars. Therefore, Mazedia teamed up with the aquarium team to designed a device using 3D animated models of a shark and a group of penguins, giving visitors the impression of interacting with these 3D animated models.

The device integrates augmented reality elements within a specific area of the aquarium, an area mostly dedicated to families or class groups composed of students and their teachers. The experience takes place in front of a great projection (5 meters wide and 2,5 meters high) in which thanks to a central camera, group members can see themselves.

For the device to properly work, visitors must respect a distance area; therefore, a line is drawn on the floor, in which groups cannot cross. Triggering the animation, once they have stepped in the right spot, virtual 3D animals appear in the projected scene, giving the impression that the animals are sharing the same space as the groups

Within the first few months, the venture was launch, the aquarium staff as well as, the visitors showed a great interest and an actual success for the device.

To make it an even more immersive experience, we have partnered with the following:

Dynamixyz a world leader agency in facial motion capture, based in Rennes, and also with two research teams: The CRENAU team (with an expertise in augmented and virtual reality and based at the Ecole Centrale de Nantes) and the PACCE team, specializing in human-machine interaction and cognition (based at the Institut Mines Télécoms of Nantes).

This partnership aims to improve the 3D animated specimens and the different interactions between the group members. Thus, the primary purpose of the article is to give an overview of the project’s means and objectives, focusing on a few outlines. First, we present the details and the limits of the existing system, followed by identifying the three main goals the project has to achieve, and finally, we will be focusing on the scientific, technical and experiential issues the consortium will have to tackle.

1. Achievements and limits of the existing device

As noted in the introduction, the existing spatial augmented reality system encourages group members to move within a designated space and see at a large projection on the wall in front of them.

The system is composed of a camera, a video projector, a series of pre-processed 3D animations of animals (a shark and a group of penguins) and a unique setting that allows a more profound sense of immersion.




The system relies on two scenarios: the shark and the group of penguins. The two animations launch periodically and have a duration of 40 seconds with about 10 seconds between each of them. The visitors set themselves in front of the projection and see themselves, a first animal will appear in the projection and gives the impression of getting closer and closer to the visitor. The 3D animations were designed to be as realistic as possible, according to the animals’ behavior within a natural setting (thus, coherent with scientific knowledge).


Also, the projective augmented reality system ( according to the taxonomy ) is technology-wise, quite simple relying on the combination of high-quality 3D animations of the animals and the immersive features of the setting. The visitor experience benefits from this very combination. As shown in the pictures, the children are sensitive to the presence of the virtual animals and will try to reach the 3D animations even though no real interaction happens between them and the animal; this gives us a good idea of the evocative power of this kind of setting. Interestingly enough, in that phase of the project, we notice that it was the visitors who adjusted and adapted their gestures and behavior to the 3D animated animals (for example, by following the penguin’s planned path to pad it on the head). The very aim of the future project is precisely to meet visitors’ existing expectations when trying to interact with the animals. Observations have shown that the children are not the only ones fascinated, also engaging adults with this evocative energy mode. (parents have been seen mimicking swimming with the shark)

This first experimentation explained to us much about how visitors reacted in the virtual presence of the 3D modeled animals and improving the system for further testing.




2. Spatial augmented reality with advanced interactions

We propose to build and improve upon the results of past experiments the different interactions between the group members and the 3D animated specimens. In order to achieve this, we must fulfill the following three challenges.

First, we want to offer a more visitor engaging experience to meet their expectations, for that to happen, we then need to provide real interactions, indicating that visitors will have real-time feedback and impact on the animal 3D model. For instance, if they try to get close to the animal, it may want to flee, depending on the species’ real behavior and habits towards humans. In addition to this kind of reaction, the position and the number of visitors in the scene will have central importance in the unfolding of the animation. The goal is to have a mutual influence on the virtual and real worlds (the behavior of the 3D animal has an impact on the visitors’ behavior and vice versa).



The second challenge is to create a favorable environment for visitors to share a common experience where interactions between visitors have a direct impact on the scenario of the animation. For example, the system can identify clusters and behavioral patterns within the group launching the animations accordingly. (adults with children, children interacting with each other, an isolated adult or child, a group of pupils with their teacher, and so on). The definition and the relevance of these clusters and patterns will constitute a work package in and of itself. To encourage the collaboration amongst the group members (school groups or family groups), integrating gamification elements as a specific but not limited only to a modality of interaction between the visitors and the virtual animations

The third challenge is to keep in mind that, even though the experimentation will take place in an aquarium, the principles can be relevant for other types of institutions and organizations, such as art museums, history museums, science centers, heritage sites, parks, and so on. This project aims to provide these kinds of players to the cultural field with the same immersive and collective experiences. For instance, in an art museum, the same technical system could allow visitors, with the help of an avatar, to either personify an existing character in a painting or to be part of the scene as a new character. This diagram gives an overview of how the systems work.


Scientific, technical and experiential issues

The proposed SAR (Spatial Augmented Reality) system at first is expected to detect group members’ position in the physical space and trigger specific movements from virtual avatars (animals). Second, based on the group members’ body and facial motion capture, adapt the 3D animation. A significant phase in achieving the goal of an immersive experience, this innovative approach produces the following issues.

First, we need to capture the position and posture of visitors in the physical space. Hence, it means that the designed SAR system can detect a group, its members, and their corporal characteristics like size and shared positions. Depth cameras like Microsoft Kinect or Intel RealSense can provide this kind of information with dedicated algorithms.

Second, the real-time detection of facial expression through their motion capture along with the definition of heuristics brings the interactive experience to the next step.

Dynamixyz has excellent expertise in this domain and will embed its technology inside the technical device. One of the technical issues in this process is to handle undesired and unexpected situations like face detection failures due to occlusion, latency or low video frame definition




Third, the project intends to describe and classify interaction triggers between the group member behavior analysis and the 3D animated animal behaviors, so far this is one of the main identified scientific issues.
Indeed, it means that on the one hand, multimodal information from sensors can be classified as corresponding to human behaviors in a database [4], and on the other side, based on these human behaviors, 3D animations are labeled with similar answers. These two classified databases include domain experts knowledge of both in human-computer interaction and cognition in etiology.

Fourth, a time-consuming technical issue is the creation of specific 3D animations. These animations should be split into the smallest units possible. Indeed, we need to reconstruct a 3D animation in the natural scene by combining many small animations in response to real-time events (humans’ actions). For the system to provide the best interactive, immersive experience, it should be flexible enough to answer group members behaviors, while delivering realistic 3D animations. Working with realistic, 3D animals is an additional challenge to provide an engaging interaction.

Finally, working with an aquarium implies specific challenges in terms of interpretation. For instance, the proposed SAR system has to deliver pedagogical messages under the form of particular animations; these messages should be triggered by specific detected behaviors, or, on demand, by the aquarium staff. Picturing the final set as both a pedagogical and a playful medium. Thus, the scenography of the physical setting will be of great significance to influence visitors’ behavior and improve the feeling and sense of immersion. The use of sound design could also be an impressive lead to follow. 

In this paper, we described a spatial augmented reality system to bring advanced interactions in the cultural sector. Our primary goal is to overcome the limitations of existing digital technologies as they are usually designed in museums as a whole. This proposal benefits from an existing setting in the aquarium of Le Croisic, France to bring the visitors to a further level of interaction with virtual specimens. By combining body and facial motion capture among the group members with databases of humans and animals’ behavior classification, we aim at providing real-time 3D animations based on the detected visitors’ behavior. This project generates many scientific and technical issues that will be investigated by a consortium of researchers and industrials in the different fields of required expertise. In a long-term perspective, we plan to demonstrate how this approach could be applied in other contexts, such as art museums, history museums or heritage sites in general. Finally, our work will investigate the consequences of such a system on interpretation strategies and collaborative experiences among groups (with a particular focus on families). This work-in-progress has been submitted to a call for proposals and is expected to start in June 2019 for a period of 24 months