In the last couple of weeks, I wrote about how I think the pandemic has affected the entertainment industry. You can read the first of the three posts here. In my third installment about the music industry, I briefly discussed that while online content has soared during the last five months, there’s also been an uptick in virtual reality concerts like the one performed by John Legend through the Wave platform. I’ve been interested in the intersection of live music and technology for some time so I figured now was as good a time as any to discuss further.
Immersive Sound
As a movie aficionado, I am all about surround sound and immersive audio. While surround sound (like the typical 5.1) “surrounds” the listener along the horizontal plane, immersive sound envelopes the listener along both the horizontal and vertical planes, creating a 3D sonic image. DTS: X and Dolby Atmos are the two main versions of immersive sound (with Dolby currently in the lead).
While immersive sound has been a fixture in movie theaters and is slowly gaining traction in high-end home theaters, it is also coming to Apple’s EarPods Pro this Fall; and last year, the Recording Academy changed one of its Grammy award categories from Best Surround Album to Best Immersive Audio Album. The applications in recorded music, Virtual Reality, and gaming are obvious. But, what about live music?
Last year, Dolby expanded its Dolby Atmos to live applications, beginning with the American Music Awards at the Microsoft Theater in LA as songs by Dua Lipa, Lizzo and Post Malone were performed in its immersive format with great success. While a lot goes into preparing and optimizing a venue for immersive sound, the possibility for a truly personal and engaging musical experience is worth further exploration.
Smartphones / Second Screen
At virtually any live event outside of the classical concert hall, smartphones are ubiquitous. Audience members take photos and video, tag the venue and performer, and comment on the experience. Facebook is getting in on the act with their new app, Venue. As they stated in their press release, “Venue aims to give fans an interactive second screen experience, curated by experts and centered on the pivotal moments of their favorite events.” Its first event was for NASCAR.
The second screen effect – watching one form of entertainment (like TV) while using a second one (like a phone or tablet) – is a real thing. In 2018, Nielsen found that approximately 45 percent of American adults “often” or “always” use a second screen while watching TV. What do you think that number is now, two years later? How about for 16-24 year-olds?
Of course for the classical concert hall, once the concert begins, cellphones are mostly verboten. Last year, for instance, violinist Anne-Sophie Mutter stopped playing a solo in the middle of a concert because an audience member had their phone out to take a picture or video. The embarrassed phone user left the hall before Ms. Mutter continued her performance. While it would be understandable if audience members were annoyed, I can imagine there must have been a better way of handling it. Can you imagine Bono stopping in the middle of the ballad “In a Little While” because the audience had their phones out taking video (which of course happens all the time)?
A small minority of orchestral organizations have been experimenting with changing that paradigm with an app called EnCue by Octava. The app allows for the creation of content that compliments the music – such as program notes, tidbits about the composer, or things to listen for – all in real time as the performance is happening. The content is sent to participating audience members who have downloaded the app. In one instance, an orchestra had special seating for those participating so as to not distract other audience members – a definite “best practice.”
This is precisely the kind of interaction/information at the moment of need that could lead to a more active listener, especially one who is used to consuming entertainment with a phone in their hand. Of course, the Field Band utilizes video screens during our shows so I can easily see the possibilities that EnCue provides. We use our screens to play videos that enhance the story-telling of the music or introduce a soloist, helping the audience make a personal connection with the soldier and the music they are about to perform.
Augmented Reality
This is a realm that I see gaining a lot of traction over the next 2-5 years. Augmented Reality, or AR, superimposes computer generated objects onto the real world, creating a “mixed” reality for the observer. Now that both Apple and Google smartphones come equipped with the capability, AR has moved into the mainstream. The two most widespread uses of AR remain the incredibly popular app Pokémon Go and real-time filters used on SnapChat.
But, we’re also seeing artists utilize AR in their live shows. U2 (which also happens to be my favorite band), has consistently pushed the envelope in an attempt to create unique experiences for, and deeper connections with, their audience – from large LED screens to creative staging – they have sought ways to break down the “fourth wall” throughout their careers. During the band’s 2018 “Experience + Innocence” tour, they used AR to enhance the audience’s experience. With the U2 Experience app, fans were able to point their phone toward the large, double-sided on-stage screen and experience the performance in a new way as it superimposed imagery onto the physical stage.
With cellphones being AR-enabled, and with the worst kept secret in tech (the Apple glasses that are rumored to be centered around AR), the sky is the limit. In the mid-term, AR could be a supplement to the traditional video screen during live performance. But there may well come a time when the quality of imagery and pervasiveness of the technology will make it so that AR becomes the norm for live performance multimedia.
Virtual Reality
While Augmented Reality superimposes computer-generated objects onto the real world, Virtual Reality (as the name implies) creates a whole new reality, blocking out the real, observed reality of the user. There have been two main applications of this for live music performance. The company/app MelodyVR takes an interesting approach to VR. Instead of creating avatars and a computer-generated space, MelodyVR creates an immersive visual experience through a multi-camera shoot, utilizing its app for iOS, Android, and VR headsets. Having watched a couple of performances with the phone app myself, it is pretty immersive, especially when wearing headphones. The observer can move around the 360 degree environment and watch the performance from different vantage points. Could it be an alternative to touring all over the country? I don’t know, but it certainly has possibilities.
MelodyVR’s product is a great way to experience a live performance “up close and personal” and make it seem like you’re right there. But, what happens if there can’t be – for whatever reason (pandemic) – a multi-camera shoot with a full house? How can artists perform for their fans? There’s more than one answer, of course, but here’s one that’s slowly gaining steam. Another avenue for Virtual Reality is the one you’re probably most familiar with – the creation of a whole new avatar-based experience. What was once science fiction is now fact.
On April 23, the developer Epic Games hosted a VR performance by rapper Travis Scott in the Fortnite virtual space for an audience of over 12 million players, and over 27 million over the three days the performance replayed. Yes, you read the right. Viewers were able to watch and listen to the performance on gaming systems such as Nintendo Switch. Epic Games sold merchandise during the 10-minute show and split the profits with the artist.
One of the leading companies in VR music is Wave, which uses a motion capture suit similar to the ones used in movies like Avengers: Endgame to create avatars that move in sync with the artist as they perform. As I wrote in my previous post, electric violinist Lindsey Sterling performed for an audience of 400,000 last year, while this past June none other than John Legend performed a 17-minute set on the Wave platform as audience members in the form of avatars (for those attending with a VR headset) or as virtual stars (for those watching on YouTube, Facebook, Twitch, and gaming platforms) interacted with Legend and donated to the charity he was supporting.
This form factor has the potential to take on a whole new level in terms of audience immersion and interactivity. Of course, this type of event will only gain traction if people can make money, especially if the idea is to replace some of the money made during a traditional tour. Exclusive merchandise, ticket tiers that get you special access and interactions…the possibilities are only limited by our imagination.
COVID-19 has taught members of the entertainment industry one main lesson: adapt or suffer the consequences. It has also hastened adaptation by those willing to be bold and unconventional. Live music will most likely always be important to people; the question is what forms will it take?
To the Color 