A First Look at Online Interactive Performance Technology

The epidemic in early 2020 swept the world, and the performance industry was greatly impacted. Many musicians and entertainment companies began to explore online concert methods. Compared with offline performances, there are many differences in the form of online live broadcasts. How to use technology to bring the relationship between audiences and performers has become a new topic. In order to better enhance the audience's sense of participation, the students of INSA HdF have some new ideas. More than 20 graduate students in the audio and video field from the school formed a group called Tempos Event and decided to add some interactive parts to the online concert. The author serves as the head of interactive technology in this group.

1. Interactive Design

As an experimental interactive performance project, the interactive design of this project is not complicated. Specifically, when the audience watching the live broadcast appears in a specific camera position, clicking on a specific area on the screen will cause the background effect to change and the lighting and video to change during the live broadcast. To give a simple example, whether the whole party will officially start from the warm-up program depends on whether a sufficient proportion of interactive users click on a specific area in the live screen in the first interactive scene. When the number of clicks meets a certain percentage, the burst animation and lighting effects rendered in real time on the background will ignite the live broadcast. There will also be different forms of interaction in the subsequent links, as well as the results of the interaction.

2. Technical realization

The technology of the interactive part mainly includes three parts: the front-end, the middle layer and the back-end. The front-end refers to the online live page and the interactive elements in the page; the back-end refers to the software for real-time rendering and the software for playing common materials; the middle layer is the bridge connecting the front-end and back-end.

When the audience clicks or hovers above the interactive element, when the trigger conditions are met, the audience will be given a direct trigger feedback animation on the browser screen, and at the same time, a successful trigger record will be added to the MySQL database of the website through Ajax. Trigger records from numerous users are processed by data analysis programs within the website and generate statistics for each interaction. At this point, the work of the website at the front end of the system is over, and the statistical data will be acquired by the middle layer.

The back-end part of this project mainly refers to the realization of video and various audio lighting effects. The part of the video is divided into two categories, one is the real-time rendering video using vvvv software, and the other is the video material produced in advance played through the Millumin software. Four projectors were used on the scene, projecting on the three facades of the curtain and the white scene in front of it, and using the built-in Mapping function of the Millumin software to model and project the curtain area and the white scene. The video rendered in real time by the vvvv software is used as an input source of Millumin, and this software method is used to replace the hardware control system like Barco Event Master E2 and the player system like WatchOut.

vvvv is an open source real-time rendering system that builds a real-time rendering workflow in a graphical way, and is an artist-friendly way to build real-time rendering programs. It can import external sources, such as video or sequence frames, or create images with its own particle, gravity, or fluid systems. In this project, most of the real-time rendering images related to interactive data required by the creative team used the software's own particle system.