Music education has now also conquered the digital space. The Konzerthaus Berlin has been exploring these technical possibilities since 2016. In this article, we provide an overview of our immersive projects.

Immersion has been the buzzword of recent years. Although the term has been around for a long time, it has only recently made the jump into numerous digital forums, blogs and analogue discussion groups. Currently, all-embracing immersion in virtual worlds, enabling one to forget the surrounding reality, is primarily associated with two technologies that have also become extremely popular: virtual reality (VR) and augmented reality (AR).

Even though the two technologies differ in their effects, applications and visualisation possibilities, they are often referred to in the same context as the media of the future. Thanks to a head-mounted display, i.e. VR glasses, VR enables the user to dive into proverbial virtual worlds. Standing on a mountain peak, flying over big cities or strolling through Versailles – thanks to VR, everything is possible from the living room couch. VR goes far beyond the 360° videos that are often uploaded to YouTube. With controllers, hand tracking and more, natural movements and a high degree of interaction with the virtual world are possible – which further enhances immersion.

Augmented reality, on the other hand, works differently. All the user needs is a powerful smartphone or AR-glasses to expand the visible reality with virtual elements. The smartphone game “Pokémon Go”, which brought AR into countless children’s rooms a few years ago, is just one example based on this technology.

How VR and AR are used depends mainly on the context. When an agency, company or cultural institution begins conceptualising a digital project, the focus should always be on content first. What exactly do we want to convey? Why did we decide to approach this digitally? What are our visitors’ media usage habits? How do we reach our users in the digital space? How and where can we offer digital projects? Only when these questions have been clarified is it time to consider which technology would be suitable. After all, technology for technology’s sake will not bring the desired added benefit.

There is a good reason for this: AR and VR have different requirements. They present content differently and the degree of interaction also differs. Here’s a simple example. A company decides to present an image gallery in the digital space. This would not do justice to the scope of VR technology – VR can do more than just display a slideshow and should be used accordingly. With AR it is quite conceivable to present images on a smartphone or tablet. But here too, the added benefit of AR must be taken into account. Does a customer want to continuously hold a smartphone to view some photos? Doesn’t it make more sense to stick with the good old medium of print? 

Employing a particular technology isn’t always a good idea. It often takes several attempts, wrong decisions and honest feedback from users to understand this. Openly dealing with the experience of failure and personal exchange with others are the best methods to find your way through the countless possibilities that immersive media have to offer and decide on the best solution for a project. And even if you are still uncertain afterwards, always have the courage to experiment!

The Konzerthaus Berlin has been placing an emphasis on innovative formats for years. It all began in 2015. Together with the HTW Berlin University of Applied Sciences, an initial 360° video was created with the Konzerthausorchester Berlin under its honorary conductor Iván Fischer. This production was shown in the vestibule of the Konzerthaus Berlin and was so well received that it led to a long-term cooperation with the HTW Berlin. The “Apollo” project, supported by the EU through the ERDF Structural Fund, got underway in 2016 and is still running today. The focus of the project is enabling classical music to be experienced in an entirely new way – and not only by the general public, but also by classical music experts. The members of the interdisciplinary project chose VR and AR for this purpose to ensure a modern-day presentation using the latest technologies. The team, coming from a diverse range of fields – programmers, developers, designers, musicologists, dramaturges and Communication Department staff at the Konzerthaus Berlin – found a common language. This was not always a matter of course in what was then still quite a new interface between classical music and technology. Programmers have to understand how the world of classical music works, musicologists have to learn what kinds of things are possible in AR or VR and what the limits of these technologies are. The mutual exchange, perspective shift and open communication have been basic requirements for the project having worked so successfully over the years.

In the following section, some of the projects introduced by the Apollo team will be described as examples of collaborative work.

About a year after the project started, the Konzerthaus Berlin published the augmented reality app “Konzerthaus Plus” in 2017. The first augmentations could be found in the 2017/18 season brochure with an interactive sound map of the Baltic States, photo galleries, musical excerpts and 3D models. Digitally enhanced postcards, an orchestra puzzle and a virtual timeline followed. Probably the most extensive project, in addition to a high-quality 3D model of the Konzerthaus and five of its halls, is the “Virtual Quartet”.

Here, members of the Konzerthaus Quartet are transformed into virtual musicians. By scanning several playing cards with the app, the respective musicians appear on the smartphone screen. Users can then decide whether they want to listen to the musicians together or concentrate on a single instrument – once a playing card is moved out of the camera’s focus, not only the virtual musician but also his or her individual instrument disappears. The Virtual Quartet is particularly suitable for music lessons in schools, as well as music education in general. Users playfully learn how a string quartet is structured and what the individual instruments are called and sound like – and can decide for themselves how they would like to explore the work being played.

To date, three major VR projects have been implemented as part of the Apollo project. The first was a multi-perspective 360° video with the Konzerthausorchester Berlin. The user takes a virtual seat on stage between the musicians and its honorary conductor Iván Fischer and can follow the interaction and non-verbal communication between the orchestra members from several perspectives. Each 360° sphere also has 3D sound, giving an authentic impression of the symphony being played.

The second project was a virtual tour through the Konzerthaus Berlin. By putting VR glasses on, the user learns more about the history and architecture of the building with the help of a narrator, pictures, videos and animations. The tour is based on 3D models of the exterior facade and the halls, which are already part of the AR app.

The third and most recent VR project is an interactive composition that was created with the support of #experimentedigital, a cultural initiative of the Aventis Foundation. Together with the American composer Mark Barden and the Mexican visual artist Julian Bonequi, Apollo members have created “Environments”, a musical score come to life that can be walked through and explored.

The complex and highly interactive project encourages users to become composers themselves by searching out and activating individual sounds in a surreal visual world.

Since the summer of 2018, the projects have been showcased in a digital exhibition that can be found in the vestibule of the Konzerthaus Berlin. The exhibition is open almost daily free of charge, offering direct, low-threshold access to everyone. Each month, volunteers overseeing the exhibition count about 10,000 visitors from all over the world – not including evening concert audiences. The applications are all communicated in German and English for international visitors. The scope of the exhibition is continually being expanded, and its content is also regularly updated. Feedback from colleagues and visitors is decisive here.

Even though the AR app “Konzerthaus Plus” is available to everyone globally and free of charge, it is an important concern of the Konzerthaus Berlin to make the digital projects available without a high level of technological knowledge in the urban realm as well. In so doing, even more people of all ages and technology levels can be reached. Thanks to the technology’s portability, Apollo members have been guests in several primary schools, using VR glasses and tablets to teach students how a symphony orchestra is organised or the sound of a string quartet. The projects are also presented at the “Kiezkonzerte” (neighbourhood concerts), which take place once a year in Berlin.

During the winter months of 2019/20, the exhibition went on its first major tour through three Berlin district libraries. For several weeks at a time, the pillars could be found in Spandau, Friedrichshain and Neukölln libraries, where they reached a completely new but very interested target group. They were accompanied by activity days and small concerts. However, the Konzerthaus Berlin also regularly opens its doors to school classes who are very interested in this modern way of learning about music. Incidentally, requests do not only come from teachers. Tourist groups and senior citizens’ associations often find their way to the vestibule and, in addition to a tour of the building, are given a guided explanation of the exhibition followed by a discussion.

The Konzerthaus Berlin embeds digital music education in its overall programme, conceptualising and communicating it in cross-media form. The projects are disseminated via social networks and thus reach a broad audience. Particularly the “Konzerthaus Plus” has extended the range of educational offerings into the digital realm. Print publications such as season brochures, festival flyers and programme booklets are also augmented to convey content in an innovative way. Konzerthaus 24/7: the logical consequence.

It is clear that despite many advantages, there are always obstacles when it comes to producing digital projects. Three cornerstones are listed here as examples (the list certainly is not exhaustive). Each project is different; sometimes more complex, sometimes more straightforward in terms of planning and implementation. The hurdles and issues that arise depend on previous experience, institutional requirements, project complexity, target group expectations and budget – just to name a few factors.

Human and financial resources

Every project costs money. This fact is both simple and understandable. However, especially when it comes to producing digital projects, many institutions find it difficult to understand how such a large sum of money could possibly turn up at the end of the cost estimate. The problem: many are familiar with top-quality animations from Hollywood films or graphics from Triple A games. But they often aren’t aware of the amount of effort involved. The request: this quality is omnipresent nowadays, so my product also needs to be high-end! The answer: of course – but only if the budget, personnel resources and timetable allow.

When a cultural institution seeks out an agency or a production company, it is of great advantage if the employees are already familiar with the client’s working environment and also the expectations of the respective target group. If not, communication is the key. After all, the project can also be implemented as a mere tech demo – whether and how it ultimately helps the institution to communicate its content and reach visitors is another question to explore.

Technological conditions vs. musical excellence

For a larger production in VR, e.g. with a symphony orchestra, the fact that classical music and technology are frequently at odds with one another should never be overlooked. Technology imposes strict requirements – the lighting on site, the distance separating the 360° cameras from the musicians and the exact positions of the microphones are just some of the aspects that need to be considered by all those involved. Then there are also the performers, whose most important goal is to make music together. Under certain circumstances, the technical conditions can lead to restrictions or altered situations in the orchestral structure. Here, too, communication between all those involved – the production company, musicians and conductor – is a decisive factor for the project’s success.

Even when a project is completed, it hasn’t ended yet. Depending on how an institution presents its project, both the software and the hardware must be maintained and serviced. In the Konzerthaus Berlin, several tablets and VR glasses are available for visitors in the digital exhibition, but the exhibition does not run itself. The hardware has to be cleaned, the app receives regular updates, questions from visitors who have never worn VR glasses and have no previous technical experience have to be answered. In the Konzerthaus Berlin, this is carried out with the help of specially trained volunteers. They support visitors at the AR pillars and encourage them to put on the VR glasses, as VR and AR are still largely unfamiliar fields in the classical field. The challenge is to sensitize visitors to digital content instead of asking them to hastily change their views.

Like many areas of life, music and listening habits are undergoing rapid technological changes. The listening experience, initially limited to the concert hall, has increasingly emancipated itself from this performance venue. Music has been recorded on vinyl records or CDs, transmitted via radio and is now available in completely digital form. Music has become omnipresent and independent; it is more immediately available than ever before. The classical music industry should see this as a challenge. Concert halls should not be afraid of new reception habits. Visitors and decision-makers have to lose the idea that digital offerings are replacing the live experience. The shared experience of a real concert cannot be replaced by VR glasses, smartphones or live streams. It is much more likely that technologies such as AR and VR will form another building block in the evolution of music communication. Linear mediation approaches are complemented by these new media possibilities. Another big plus point: AR, VR, but also apps and games in general, ensure enormously low-threshold access. Nearly every teenager has a smartphone that he or she can use to access mediated content; it is an everyday part of their world. A smartphone is also portable, so it can be used anywhere and at any time. Cultural participation has never been so omnipresent, direct and simple as it is today – if cultural institutions understand this as an opportunity, both for themselves and the upcoming generation of classical music lovers, and use it in a meaningful and lasting way. The potential of digital cultural mediation, which is still in its infancy in many places, is palpable. It creates new media-based access to knowledge and education.