artificial intelligence
Page: 18
Lawyers for the RIAA are aiming to shut down a popular Discord server centered on artificial intelligence and voice models, the latest effort by music companies to rein in the disruptive new technology.
In an action filed last week in D.C. federal court, attorneys for RIAA obtained a subpoena demanding that Discord reveal the identities of users on “AI Hub,” a message board with 145,000 members that calls itself “a community dedicated to making AI voices and songs.”
In a letter to Discord presenting the company with the subpoena, the RIAA said those users had “infringed … copyrighted sound recordings” and that the tech company was required to hand over names, physical addresses, payment info, IP addresses and other identifying details.
The group’s lawyers also sent Digital Millennium Copyright Act takedown notices to Discord, first in late May and then again next week. The group demanded that Discord disable access to the server, remove or disable the infringing material, and inform the server’s users “of the illegality of their conduct.”
“This server [is] dedicated to infringing our members’ copyrighted sound recordings by offering, selling, linking to, hosting, streaming, and/or distributing files containing our members’ sound recordings without authorization,” the RIAA’s lawyers wrote in their June letter to Discord, which was obtained by Billboard. “We are asking for your immediate assistance in stopping this unauthorized activity.”
The subpoena against Discord was obtained under the DMCA’s Section 512(h), which enables rights holders like the RIAA’s members to unmask the identities of anonymous online infringers in certain circumstances.
Discord can fight back by seeking to “quash” the subpoena; Twitter won such a challenge last year, when a federal judge ruled that the First Amendment rights of a user trumped the need for an unmasking order. It could also refuse to honor the takedown, but that would put the site itself at risk of litigation.
As of Thursday evening (June 22), the main AI Hub server remained up on Discord; it was unclear if individual content or sub-channels had been removed. A spokesperson for the company did not return a request for comment.
In a statement to Billboard, an RIAA spokesperson confirmed that the group had taken the action against AI Hub. “When those who seek to profit from AI train their systems on unauthorized content, it undermines the entire music ecosystem – harming creators, fans, and responsible developers alike. This action seeks to help ensure that lawless systems that exploit the life’s work of artists without consent cannot and do not become the future of AI.”
The RIAA’s actions are just the latest sign that the explosive growth of AI technologies over the past year has sparked serious concerns in the music industry.
One big fear is that copyrighted songs are being used en masse to “train” AI models, all without any compensation going to the songwriters or artists that created them. In April, Universal Music Group demanded that Spotify and other streaming services prevent AI companies from doing so on their platforms, warning that it “will not hesitate to take steps to protect our rights.”
Another fear is the proliferation of so-called deepfake versions of popular music, like the AI-generated fake Drake and The Weeknd track that went viral in April. That song was quickly pulled down, but its uncanny vocals and mass popularity sparked concerns about future celebrity rip offs.
For RIAA, AI Hub likely triggered both of those worries. The server features numerous “voice models” that mimic the voices of specific real singers, including Michael Jackson and Frank Sinatra. And in the wake of the RIAA’s actions, users on the Discord server speculated Thursday that the takedowns were filed because users had disclosed that some of the models had been trained on copyrighted songs.
“We have had certain threats from record labels to takedown models, mainly because some posters decided to share datasets full of copyrighted music publicly,” one AI Hub admin wrote. “If you want to avoid unnecessary takedowns[,] most importantly, do NOT share the full dataset if you have copyrighted material in the dataset. The voice model itself is fine, but don’t share the dataset.”
As the music industry continues to grapple with the popularity and advancement of AI, some find themselves trying to figure out how to ethically incorporate artificial intelligence into their work. Earlier this month, when music titan Paul McCartney shared that he would be implementing AI in this process of finishing a new as-yet-untitled Beatles song, […]
Calling the rapid growth of artificial intelligence tools a “moment of revolution,” Senate Majority Leader Chuck Schumer said Wednesday that the government must act quickly to regulate companies that are developing it.
The New York Democrat said he is working on what he calls “exceedingly ambitious” bipartisan legislation to maximize the technology’s benefits and mitigate significant risks.
While Schumer did not lay out details of such legislation, he offered some key goals: protect U.S. elections from AI-generated misinformation or interference, shield U.S. workers and intellectual property, prevent exploitation by AI algorithms and create new guardrails to ward off bad actors.
AI legislation also should promote American innovation, Schumer said in a speech at the Center for Strategic and International Studies, a Washington think tank.
“If applied correctly, AI promises to transform life on Earth for the better,” Schumer said. “It will reshape how we fight disease, tackle hunger, manage our lives, enrich our minds and ensure peace. But there are real dangers that present themselves as well: job displacement, misinformation, a new age of weaponry and the risk of being unable to manage this new technology altogether.”
Schumer’s declaration of urgency comes weeks after scientists and tech industry leaders, including high-level executives at Microsoft and Google, issued a warning about the perils that artificial intelligence could pose to humankind.
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” their statement said.
Worries about artificial intelligence systems outsmarting humans and running wild have intensified in recent months with the rise of a new generation of highly capable AI chatbots such as ChatGPT. It has sent countries around the world scrambling to come up with regulations for the developing technology, with the European Union blazing the trail with its AI Act expected to be approved later this year.
On Tuesday, President Joe Biden convened a group of technology leaders in San Francisco to debate what he called the “risks and enormous promises” of artificial intelligence. In May, the administration brought together tech CEOs at the White House to discuss these issues, with the Democratic president telling them, “What you’re doing has enormous potential and enormous danger.”
“We’ll see more technological change in the next 10 years that we saw in the last 50 years,” Biden said.
White House chief of staff Jeff Zients’ office is developing a set of actions the federal government can take over the coming weeks regarding AI, according to the White House.
Schumer’s hands-on involvement in crafting AI legislation is unusual, as Senate leaders usually leave the task to individual senators or committees. But he has taken a personal interest in regulating the development of artificial intelligence, arguing that it is urgent as companies have already introduced human-like chatbots and other products that could alter life as we know it. He is working with another Democrat, Sen. Martin Heinrich of New Mexico, and Republican Sens. Mike Rounds of South Dakota and Todd Young of Indiana to speak with experts, educate colleagues and write the legislation.
It’s an unexpected role for Schumer, in particular, who famously carries a low-tech flip phone, and for the Senate as a whole, where the pace of legislation is often glacial.
Senators average around retirement age and aren’t known for their mastery of high-tech. They’ve been mocked in recent years for basic questions at hearings — asking Facebook founder Mark Zuckerberg simple questions about how his platform works at a 2018 hearing on Russian interference, for example — and for a bipartisan reluctance to regulate the technology industry at all.
Schumer, along with several Republican colleagues, say the federal government can no longer afford to be laissez-faire with tech companies.
“If the government doesn’t step in, who will fill its place?” Schumer asked. “Individuals and the private sector can’t do the work of protecting our country. Even if many developers have good intentions, there will always be rogue actors, unscrupulous companies, and foreign adversaries that seek to harm us. And companies may not be willing to insert guardrails on their own, certainly if their competitors are not required to insert them as well.”
Attempting to regulate AI, Schumer said, “is unlike anything Congress has dealt with before.”
It is unclear if Schumer will be able to accomplish his goals. The effort is in its earliest stages, with the bipartisan working group just starting a series of briefings for all 100 senators to get them up to speed. In the House, legislation to regulate or oversee artificial intelligence has been more scattershot, and Republican leaders have not laid out any ambitious goals.
Schumer acknowledged that there are more questions than answers about the technology.
“It’s not like labor or healthcare or defense where Congress has a long history we can work off of,” Schumer said. “In fact, experts admit nobody is even sure which questions policymakers should be asking. In many ways, we’re starting from scratch.”
The first time Ashley Elzinga, a 33-year-old DJ from Traverse City, Mich., heard her doppelgänger’s voice, she was not happy. Not because the sound of an artificial-intelligence imposter was so eerie. Not because AI technology portends robots might someday replace her. She didn’t like the way AI Ashley pronounced “news.” “I was like, ‘Why are they saying nooooose?’” recalls Elzinga in her flat Midwestern accent. “I was so embarrassed.”
It took a few tries for the engineers at Futuri Media, a Cleveland-based AI specialist, to find the right vocal balance between flat and sharp, deadpan and excited. “Now she’s ironing out…,” says Elzinga, a midday host for Portland Top 40 station Live 95.5, then corrects herself: “Now it’s ironing out…. She, or it, is starting to have more emotion and be a bit more accurate resemblance.”
AI Ashley, as Live 95.5 refers to the cloned voice on the air, made “her” debut on the air last Friday, delivering news, introducing songs and hyping station promotions in alternating speaking segments with the real Ashley Z, as Elzinga is known. Live 95.5 hasn’t received any listener complaints, says Dylan Salisbury, the station’s content director: “I don’t even know if they realize it yet.”
Alpha Media, owner of Live 95.5, started experimenting with AI voice technology last fall, according to Phil Becker, the company’s executive vp of content. When company execs learned Elzinga was about to take the full-time job in Traverse City, potentially reducing her hours on Live 95.5, they saw her as a “perfect storm” case study for an on-air test, he says: “The line in Moneyball is ‘the first guy through the wall always gets bloodied.’ That’s where we are right now. We’re OK playing some Moneyball-style radio, because it wins championships.”
Elzinga and Salisbury see AI as an efficiency tool, a way of stretching DJs’ hours so listeners can hear their voices even when they’re not physically present. For Elzinga, who multi-tasks her way through a full-time morning-show gig at her hometown Top 40 station WKHQ, then “tracks” her voice remotely for Live 95.5 and another station in Seattle every day, AI Ashley allows her to work even more. She owns the rights to her voice, approves every on-air AI usage and, Salisbury says, “We have increased her fee.”
“If she says stop, we have to stop,” Salisbury adds. “We’re trying to be respectful during the wild West of AI and go where we think the law is going to go.”
We made history as the world’s first radio station with an AI DJ! Our midday host Ashley has become AI Ashley! We can’t wait for you to meet Ashley, the world’s first artificially intelligent DJ. As to the intelligence of our other DJ’s…we’ll save that for another post 😉 pic.twitter.com/CtlMhYU0IO
— Live 95.5 (@live955) June 13, 2023
Of course, what is a neat, little, high-tech, mostly risk-free magic trick for Elzinga, Salisbury and Alpha Media, the Portland broadcast company that owns Live 95.5 and 205 other stations, is a terrifying prospect for much of the radio industry. When the station posted excitedly about AI Ashley last week, Twitter erupted: An NPR host tweeted an “everyone disliked that” meme, a freelance writer wanted to know, “Why would you participate in the very public elimination of your job?” and even J. Smith-Cameron, who plays Gerri on HBO’s Succession, wondered if Elzinga was “worried you’ll have ALL the days off now that they cloned you?”
For the past three decades, the broadcast industry has faced consolidation and extreme cost-cutting that has oftentimes meant layoffs of on-air talent. Over the past few years, DJs for local radio shows have been outsourced from other markets — much like Elzinga does in Portland and Seattle from her home in Michigan.
“They are eagerly stripping away, as fast as they can, the thing that makes radio unique,” says former radio host and station manager Michele Coppola, who’s now a Portland copywriter.
“My fear is there will be some owners that will [say], ‘This is an efficiency, this is a way for us to save money — that will further deplete the body of truly talented radio people,” adds Steve Reynolds, a Raleigh, N.C., talent coach for radio personalities.
“Futuri claims it’s going to be a tool, just like any other tool, to make a job easier,” says Lance Venta, owner and publisher of Radio Insight. “Voice-tracking, when used properly, is a tool. When it’s used to have a talent voice 35 stations to save money, it’s no longer a tool — it’s a weapon.”
Radio Waits
So far, the rest of the U.S. broadcast industry has yet to plunge into on-air AI voices as aggressively as Live 95.5. But radio stations around the world, and their digital competitors, have tinkered with the technology – and have suggested they may expand. In April, a Swiss station used AI to clone five human presenters on the air; comedian Drew Carey used an AI-generated approximation of his voice on his SiriusXM show in March; and in February, Spotify launched a (voiceless) AI-curated, personalized broadcast called “DJ.” During an April conference call about a soft advertising market, Bob Pittman, chairman and CEO of top broadcaster iHeartMedia, told investors after a 3.8% drop in revenue, “We and every other company are looking at how to use AI. I think AI can fundamentally change the cost structure of the company.”
At Audacy, the second-biggest broadcaster, execs have done a “fair bit of experimentation” with AI tools, from voice applications to ChatGPT-style generative text that helps produce drafts of advertising scripts, according to Jeff Sottolano, executive vp and head of programming. But he’s not convinced an AI Ashley-style experiment has “value it creates for the consumer,” because Alpha Media had to expend “up-front investment” on training, reviewing, post-production and editing — all of which, at least for now, contradict the company’s efforts for greater efficiency and cost-cutting.
“All that said, I expect it will continue to get better and easier and faster,” he says. “We don’t look at this as something that’s finished, but something that’s going to continue to evolve. Just because we haven’t done it today doesn’t mean we might not do it tomorrow.”
The human Ashley is happy with the AI arrangement as long as she and her robot counterpart are clearly identified as “Ashley Z” or “AI Ashley” every time she — or it — appears on the air. “You just need to make sure integrity comes first,” she says.
LONDON — Amid increasing concern among artists, songwriters, record labels and publishers over the impact of artificial intelligence (AI) on the music industry, European regulators are finalizing sweeping new laws that will help determine what AI companies can and cannot do with copyrighted music works.
On Wednesday (June 14), Members of the European Parliament (MEPs) voted overwhelmingly in favor of the Artificial Intelligence (AI) Act with 499 votes for, 28 against and 93 abstentions. The draft legislation, which was first proposed in April 2021 and covers a wide range of AI applications, including its use in the music industry, will now go before the European Parliament, European Commission and the European Council for review and possible amendments ahead of its planned adoption by the end of the year.
For music rightsholders, the European Union’s (EU) AI Act is the world’s first legal framework for regulating AI technology in the record business and comes as other countries, including the United States, China and the United Kingdom, explore their own paths to policing the rapidly evolving AI sector.
The EU proposals state that generative AI systems will be forced to disclose any content that they produce which is AI-generated — helping distinguish deep-fake content from the real thing — and provide detailed publicly available summaries of any copyright-protected music or data that they have used for training purposes.
“The AI Act will set the tone worldwide in the development and governance of artificial intelligence,” MEP and co-rapporteur Dragos Tudorache said following Wednesday’s vote. The EU legislation would ensure that AI technology “evolves and is used in accordance with the European values of democracy, fundamental rights, and the rule of law,” he added.
The EU’s AI Act arrives as the music business is urgently trying to respond to recent advances in the technology. The issue came to a head in April with the release of “Heart on My Sleeve,” the now-infamous song uploaded to TikTok that is said to have been created using AI to imitate vocals from Drake and The Weeknd. The song was quickly pulled from streaming services following a request from Universal Music Group, which represents both artists, but not before it had racked up hundreds of thousands of streams.
A few days before “Heart on My Sleeve” become a short-lived viral hit, UMG wrote to streaming services, including Spotify and Apple Music, asking them to stop AI companies from accessing the label’s copyrighted songs “without obtaining the required consents” to “train” their machines. The Recording Industry Association of America (RIAA) has also warned against AI companies violating copyrights by using existing music to generate new tunes.
If the EU’s AI Act passes in its present draft form, it will strengthen supplementary protections against the unlawful use of music in training AI systems. Existing European laws dealing with text and data-mining copyright exceptions mean that rightsholders will still technically need to opt out of those exceptions if they want to ensure their music is not used by AI companies that are either operating or accessible in the European Union.
The AI Act would not undo or change any of the copyright protections currently provided under EU law, including the Copyright Directive, which came into force in 2019 and effectively ended safe harbor provisions for digital platforms in Europe.
That means that if an AI company were to use copyright-protected songs for training purposes — and publicly declare the material it had used as required by the AI Act — it would still be subject to infringement claims for any AI-generated content it then tried to commercially release, including infringement of the copyright, legal, personality and data rights of artists and rightsholders.
“What cannot, is not, and will not be tolerated anywhere is infringement of songwriters’ and composers’ rights,” said John Phelan, director general of international music publishing trade association ICMP, in a statement. The AI Act, he says, will ensure “special attention for intellectual property rights” but further improvements to the legislation “are there to be won.”
More than six decades after their formation, Paul McCartney says the final-ever Beatles song is on its way thanks to the miracle of modern technology. Speaking to BBC Radio 4’s Today, Sir Paul said that he has been using artificial intelligence to “extricate” John Lennon’s voice from an old demo to complete the untitled track.
“We just finished it up and it’ll be released this year,” he said, of the untitled song that the BBC speculated could be a 1978 Lennon composition called “Now and Then.” The single was reportedly in the running to serve as a “reunion song” for the 1995 Anthology series, which included two new songs based on demos recorded by Lennon after the group split, 1995’s “Free As a Bird” and 1996’s “Real Love,” produced by ELO’s Jeff Lynne. Those tracks were the first “new” Beatles” releases in more than 25 years.
McCartney reportedly received the demo for the new track from Lennon’s widow, Yoko Ono, in 1994; the song was one of several on a cassette labelled “For Paul” that Lennon made shortly before his murder in 1980. The BBC reported that the tracks were “lo-fi and embryonic” and mostly recorded on a boombox by Lennon on a piano in his New York apartment.
The BBC reported that the living members of the band tried to record the “apologetic” love song “Now and Then” around the time of the Anthology release, but abandoned the sessions in short order. “It was one day — one afternoon, really — messing with it,” Lynne said. “The song had a chorus but is almost totally lacking in verses. We did the backing track, a rough go that we really didn’t finish.”
McCartney later said guitarist/singer George Harrison refused to work on “Now and Then,” saying the sound quality on Lennon’s vocals was “rubbish… George didn’t like it. The Beatles being a democracy, we didn’t do it.” The BBC reported that there were also reportedly technical issues with the original, due to some persistent “buzz” from the electrical circuits in Lennon’s apartment. The new version of the demo reportedly popped up on a bootleg CD in 2009, minus the background noise.
In a 2012 BBC documentary on Lynne, McCartney said, “that one’s still lingering around… so I’m going to nick in with Jeff and do it. Finish it, one of these days.” And while it is still unknown if that song is the one due out, the BBC reported that technical advances employed during the making of Peter Jackson’s Get Back Beatles documentary series — during which dialog editor Emile de la Rey trained computers to recognize the Beatles’ voices and separate them from background noise, including their own instruments — allowed the team to create “clean” audio. That same technology also allowed McCartney to sing a virtual duet with Lennon on his most recent tour.
“He [Jackson] was able to extricate John’s voice from a ropey little bit of cassette,” McCartney told Radio 4 in explaining how the tech used in the documentary helped him work on the “new” song. “We had John’s voice and a piano and he could separate them with AI. They tell the machine, ‘That’s the voice. This is a guitar. Lose the guitar.’ So when we came to make what will be the last Beatles’ record, it was a demo that John had [and] we were able to take John’s voice and get it pure through this AI. Then we can mix the record, as you would normally do. So it gives you some sort of leeway.”
At press time a release date for the Beatles track had not been announced.
LONDON – Key players from the United Kingdom’s music industry have pledged to improve the digital metadata for song recordings to help ensure that artists and rightsholders are paid fairly.
Under the voluntary two-year agreement published Wednesday (May 31) by the U.K. government’s Intellectual Property Office (IPO), the U.K. industry commits to progressively improve metadata for new music recordings and to accurately award credits on streaming services.
Signatories include representatives of major and independent record labels, publishers, creators and streaming services. Trade bodes BPI —which represents more than 500 labels, including the U.K. arms of Universal Music Group, Sony Music Entertainment and Warner Music Group — and the Association of Independent Music (AIM), which acts on behalf of U.K. independent record labels and music companies, are among the music groups backing the pledge
The U.K. government also says that it will be setting up a new working group with music business stakeholders “to explore and consider industry-led actions on remuneration for existing and future creators” over the coming weeks. The working group will review a range of issues facing creators and rightsholders, says the IPO, including the impact of artificial intelligence.
“The U.K. is a hotbed for world-beating musical talent but as technology advances we need our thriving music industry to continue to offer viable career opportunities,” John Whittingdale, a U.K. government minister for the creative industries, says in a statement.
Whittingdale calls the metadata pact a “landmark agreement” that will help ensure musicians “in the digital age are fairly credited and compensated for their contributions and creativity.”
As streaming has grown to become the dominant mode of music consumption globally, the need for accurate metadata — the descriptive data underlying music recordings detailing who contributed what to a song’s creation and who owns it — has grown in importance in determining who gets paid what share of a track’s streaming royalties. But digital platforms, artists and rightsholders have complained that streaming metadata is often incomplete or inaccurate, which delays payments to songwriters and labels.
“Currently a fraction of recordings being streamed properly credit the people that wrote and published the song,” says Graham Davies, CEO of U.K. songwriters trade group The Ivors Academy, which has also signed the metadata agreement.
In 2021 the Ivors Academy estimated that at least £500 million ($625 million) in global streaming royalties a year were being wrongly allocated or not being paid at all due to missing or incomplete metadata. In a statement, Davies says the pledge “gives us the first proper step on a journey of closing the metadata gap.”
Sophie Jones, BPI interim CEO, says she hopes the agreement will lead to “immediate improvements” in the “speed and accuracy of songwriter payments.”
Other music groups and companies backing what is officially titled the United Kingdom Industry Agreement on Music Streaming Metadata include The Digital Entertainment and Retail Association (ERA), whose members include streaming services; Hipgnosis Songs Fund; the Music Publishers Association (MPA); Musicians’ Union (MU); Featured Artists Coalition (The FAC) and U.K. collecting societies PRS for Music and Phonographic Performance Limited (PPL).
Wednesday’s metadata agreement is the latest in an ongoing series of government-led interventions into the U.K. music industry that spun out of a Parliamentary inquiry into the economics of the music streaming business.
That initial review concluded in July 2021 with the Department for Digital, Culture, Media & Sport (DCMS) Select Committee calling into question the major record labels’ dominance of the industry and branding the global streaming model as unsustainable in its current form, saying it “needs a complete reset.”
Since then, the government has initiated multiple investigations and inquiries into the U.K. music industry, including a market study review of the record business by the U.K. competition regulator. That ended in November when the Competition and Markets Authority (CMA) determined that low returns from streaming for most artists “are not the result of ineffective competition” between the three major labels.
Last year the British government set up two working groups – led by the IPO and made up of industry stakeholders — looking at issues raised by the Parliament probe. One focused on metadata, which has now concluded. A second working group is looking at improving transparency around licensing and royalties for artists. The IPO says it is due to publish the second group’s findings, including a code of practice, in the coming months.
Earlier this year, Oleg Stavitsky, co-founder/CEO of Endel, laid out a vision for how his company’s AI-driven functional soundscapes could help the major labels — even as anxiety around AI was reaching new heights. “We can process the stems [the audio building blocks of a track] from Miles Davis’ Kind of Blue and come back with a functional sleep version of that album,” Stavitsky told Billboard. At the time, he said his company was in talks with all the major labels about this possibility.
A few short months later, Stavitsky will have a chance to do exactly that: Endel announced a new partnership with Universal Music Group on Tuesday (May 23). In a statement, Endel’s CEO said his company will put “AI to work and help UMG build new and exciting offerings to promote wellness and banish the perceived threat around AI.”
“Our goal was always to help people focus, relax, and sleep with the power of sound,” Stavitsky added. “AI is the perfect tool for this. Today, seeing our technology being applied to turn your favorite music into functional soundscapes is a dream come true.” Artists from Republic and Interscope will be the first to participate — though the announcement omitted any names — with their soundscapes arriving “within the next few months.”
Endel focuses on creating “sound that is not designed for conscious listening,” Stavitsky told Billboard earlier this year. “Music is something you consciously listen to when you actually want to listen to a song or an album or a melody,” he explained. “What we produce is something that blends with the background and is scientifically engineered to put you in a certain cognitive state.”
Endel’s technology can spit out these soundscapes effectively at the click of a button. “The model is trained using the stems that are either produced in-house by our team, led by co-founder and chief composer Dmitry Evgrafov (who’s himself an established neo-classical artist), or licensed from artists that we’ve worked with,” Stavitsky said. “The trick is all of the stems” — Endel has used stems from James Blake, Miguel and Grimes —”are created following the scientific framework created by our product team in consultation with neuroscientists.”
Some people in the music industry have taken to calling sounds designed for sleep, study, or soothing frayed nerves “functional music.” And while it maintains a low profile, it’s an increasingly popular and lucrative space. “Science tells us that nature sounds and water sounds have a calming effect on your cognitive state,” Stavitsky noted this winter. “So naturally, people are turning to this type of content more and more.”
Early in 2022, Endel estimated that the size of the functional music market is 10 billions streams a month across all platforms. (The company has since raised its estimate to 15 billion streams a month.) If true, that would mean functional music is several times more popular than the biggest superstars. “Every day, hundreds of millions of people are self-medicating with sound,” Stavitsky wrote in March. “If you look at the top 10 most popular playlists at any major streaming service, you’ll see at least 3-4 ‘functional’ playlists: meditation, studying, reading, relaxation, focus, sleep, and so on.”
But this has caused the music industry some concern. Major labels have not historically focused on making this kind of music. Most streaming services pay rights holders according to their share of total plays; when listeners turn to functional music to read a book or wind down after a long day, that means they’re not playing major label artists, and the companies make less money. In a memo to staff in January, UMG CEO Lucian Grainge complained that “consumers are increasingly being guided by algorithms to lower-quality functional content that in some cases can barely pass for ‘music.’”
But record companies can’t eliminate listener demand for functional music. It makes sense, then, that they would try to take over a chunk of the market. And Stavitsky has been savvy, actively pushing Endel’s technology as a way for the labels to “win back market share.”
Back in 2019, Endel entered into a distribution agreement for 20 albums with Warner Music Group. And the company announced its new partnership with UMG this week. In a statement, Michael Nash, UMG’s evp and chief digital officer, praised Endel’s “impressive ingenuity and scientific innovation.”
“We are excited to work together,” Nash continued, “and utilize their patented AI technology to create new music soundscapes — anchored in our artist-centric philosophy — that are designed to enhance audience wellness, powered by AI that respects artists’ rights in its development.”
How magical would it be if we listened to music and music listened back to us?” asks Philip Sheppard, the co-founder/CEO of Lifescore, a U.K. startup that creates soundtracks tailored to users’ functional needs, from sleep to focus to fitness.
Though the premise sounds like science fiction, a number of new companies are already working with technology that attunes music to listeners’ movements in video games, workouts, virtual reality — even the aesthetics of their Snapchat lenses. Much as a film composer accentuates pivotal moments in the story with perfectly timed swells and crescendos, these innovations are being used to create bespoke soundtracks in real time.
One of the most fertile testing grounds for “dynamic” or “personalized” music, as it is called, is the gaming sector. Gamers tend to be avid music fans who stream songs an average of 7.6 hours a week — more than double the rate of the average consumer, according to MIDiA Research — and for some time now, game developers have linked players’ in-game movements to changes in lighting, setting and other parameters to enhance their storytelling.
David Knox, president of Reactional Music, the maker of an interactive music engine for video games, says “the final frontier for innovation in gaming is music.” Until recently, video-game music has consisted of loop-based scores or licensed tracks. Because of its repetitiveness, Knox says many users mute game soundtracks in favor of Spotify or Apple Music.
To compete with this, Knox says Reactional’s “rules-based music engine” applies the same reactive technology used in gaming graphics to the soundtrack, enabling, for example, a player in a first-person-shooter game to fire a gun in time with the beat of a song. As the technology evolves, Knox says soundtracks could transform to reflect the state of a player’s health or the level of danger.
This same technology could work with augmented reality and the so-called metaverse. Minibeats, a company that creates social media lenses with interactive musical elements, is in the process of incorporating dynamic music technology, which it calls “musical cameras,” into its AR filters for Snapchat. For one of its first launches, Minibeats partnered with Rhino and Stax Records in February to promote the 30th anniversary of the release of Booker T. & The M.G.’s’ “Green Onions.” One Minibeats filter turns users’ heads into green onions and allows them to control when the song’s signature Hammond organ riff courses through body and facial movements. Another filter morphs users’ faces into spinning vinyl records, allowing them to control when the song’s guitar and keys start and stop by opening and closing their mouths.
When imagining the future of dynamic music, Mike Butera, the company’s founder and CEO, says Disney’s Fantasia comes to mind. The ambitious 1940 film, which mixes animation and live action and features Mickey Mouse in his iconic sorcerer’s hat, syncs vibrantly colored dream-like visuals with a score that enhances what’s transpiring onscreen. “Imagine if we transformed your day-to-day life into something like that?” Butera says. “The mundanity of drinking coffee, walking the dog and driving to work [turned] into something [that] can be soundtracked with your own personal score that you control, whether that’s through a phone camera or AR glasses.”
These startups all claim that they have received only glowing feedback from the music business so far, and many have formed key partnerships. Hipgnosis recently announced a deal with Reactional Music to help bring its catalog of hit songs to the startup. Bentley and Audi have made deals with Lifescore to get dynamic soundtracks into cars, and Warner Music Group counts itself as an investor as well. Minibeats says it’s “in discussion with all the major labels,” though beyond its Rhino-Stax partnership, the company declined to disclose more details.
These emerging capabilities are typically powered by artificial intelligence to adapt recorded music to malleable experiences, but unlike other AI companies trying to create machine-made music with the touch of a button, these dynamic music startups either license preexisting, human-made songs or commission composers to create new or more dynamic compositions.
Lifescore pays composers to record a number of separate elements of a song, known as “stems,” and then, Sheppard says, its technology works with the resulting audio files like “shuffling a deck of cards,” assembling newfound arrangements in configurations intended to support a user’s need for focus while studying or working, for example, or sleep.
In the case of preexisting tracks, companies like Minibeats partner with Audioshake, a firm that uses AI to break down songs into individual, standardized stems, so that they can easily manipulate a song’s instrumental mix — guitar, drums, vocals, etc. — in real time. Audioshake’s proprietary technology is especially helpful in the case of older recordings in which the copyright owner no longer has the stems.Audioshake founder/CEO Jessica Powell says one reason she thinks the music industry has embraced this innovation is its potential to spur music discovery. “I think the same way TikTok pushes new songs, gaming — as well as other use cases — have enormous potential to introduce people to music,” whether that be a catalog track or a new release.
Though this technology is new, interactivity has been long seen as a way to create powerful bonds between fans and songs. Powell points to popular video games like Guitar Hero and Rock Band as successful examples. Karaoke is another. One could even point to the more recent novelty of stem players, like those Ye peddled during the release of his album Donda 2, as a way of engaging listeners. At a time when much of music discovery is passive — scrolling TikTok, streaming an editorial playlist or turning on the radio — musical interactivity and now personalization promises a stronger bond.
Knox at Reactional says interactive music also has economic potential. In-game purchases — which allow players to buy customizable elements like cars, weapons and outfits — dwarfed global recorded-music revenue in 2020, with players spending $97 billion in-game compared with music’s $35.9 billion (retail values), according to MIDiA Research. “In the same way you put hooks into a game, allowing someone to pay to change their appearance at a certain point, a game developer working with us could create a hook that unlocks access to the Reactional platform, letting players buy their favorite songs,” he says.
Since at least the advent of the transistor radio, consumers have used music to soundtrack their lives, but until recently, personalization of those soundtracks was limited to song selection and playlist curation. The songs themselves were unchangeable. Those on the forefront of dynamic music contend that it marries recorded music with the goose bumps-inducing, real-time experience of listening to something live.
“You know how you listen to a live performance, and the musicians are influenced by your energy in the room?” asks Erin Corsi, director of communications for Lifescore. “That’s what this is. Though this also feels like something new, it feels like we are finally able to get back to how music started.”
Beatdapp co-founders and co-CEOs Morgan Hayduk and Andrew Batey were not initially focused on fighting streaming manipulation. Batey spent years in digital marketing, while Hayduk formerly worked as a lobbyist for the Canadian music industry in the area of copyright protection. At first, they teamed to build an auditing tool that would enable labels to evaluate inconsistencies between their sales reports and streaming services’ server logs. Conversations with label executives indicated that “there were pretty often material discrepancies,” Hayduk says.
As he and Batey tried to understand those inconsistencies, it became clear that streaming manipulation was causing some of them, and Beatdapp embarked on developing a tool to detect fraudulent streams — which Hayduk defines as the leveraging of “bots, stolen accounts or manipulated platform features” to steal streaming income — and prevent them from impacting payouts.
According to a recent report from the Centre National de la Musique (CNM), a government-backed organization that supports France’s music industry, in 2021, over 1 billion music streams — between 1% and 3% of all streams generated in the country that year — were fraudulent. “The methods used by fraudsters are constantly evolving and improving,” the report noted, “and fraud seems to be getting easier and easier to commit.” If that percentage was applied to IFPI’s estimate that global streaming revenue totaled $17.5 billion in 2022, fraudulent streams would amount to $350 million in potential lost income for legitimate rights holders.
Beatdapp’s software sifts through massive amounts of data from partners — including labels, distributors and streaming services — to identify and investigate suspicious patterns. In one case, it identified 10,000 accounts all playing the same 63 tracks. The pair say the company now analyzes hundreds of billions of streams, and while they declined to identify partners, they recently started working with SoundCloud and Napster, according to two sources.
“If we can make this industry less attractive for financial fraudsters, that will make a positive difference for everybody who’s working on music,” Hayduk says. “That’s what animates us.”
Why is streaming fraud an important issue?
MORGAN HAYDUK: It hurts everyone who makes a living in the music industry and, left unchecked, creates this promotional race to the bottom where everyone believes they have to cheat to succeed. In cybersecurity terms, it’s important to shrink the attack surface of the industry.
ANDREW BATEY: In an industry where it’s already hard to make something and then promote something and then get paid, you should at least get paid correctly.
How much data is Beatdapp analyzing at this point?
HAYDUK: We’re looking at about 320 billion streams now. That’s about 13 trillion individualized streaming data points when you account for all of the metadata associated with each of those streams. We expect to add data in the neighborhood of another 50 billion streams in [the second quarter] and about another 2 trillion data points on that.
BATEY: It’s not just the individual stream. You might make 12 decisions in an app, such as how you search — if you clicked on the artist first and then you looked at their song list. We’re capturing all of that, anonymized across users. All of that context helps us because if somebody consistently hits, let’s say, the exact 11 things for every song they play, that’s a pretty obvious case of fraud if they’ve done that 3,000 times in a week.
How has the industry’s perception of streaming fraud changed since you started Beatdapp?
HAYDUK: Just hearing people acknowledge the issue is probably the biggest shift. It used to be verboten to speak publicly about streaming fraud. It was all behind closed doors. But I don’t think you can fix a problem until you accept its existence. We’re starting to get there now and [are] seeing a more widespread willingness to put in place solutions.
How has your perception of the problem changed as your data set has expanded?
HAYDUK: The biggest revelation to us has to be that this is way closer to death by a thousand paper cuts than it is a top-of-the-market problem. If you asked us where most of the fraud came from 18 months ago, we probably would have pointed the finger at bigger artists because we would have thought they had the most to gain. But we were missing the point of most of this activity. It’s not about changing perception; it’s about making money. This isn’t a phenomenon that’s driven by major labels and major independent label artists or their top artists. The overwhelming majority, like upwards of 80% of what we see is fraud, is coming from — call it non-music content. It’s not being released for popular consumption or because these are artists who are trying to get noticed. These are releases that have no commercial purpose except as [instruments of] fraud.
BATEY: When we first started, we genuinely thought fraud would be 1% to 3%. Now we think it’s closer to 10% [though some of this is caught]. Also we would have guessed that most of the fraud would occur on the platforms where people were — Spotify, Apple, YouTube. But because it’s a lot of financially motivated fraud, what we actually see is that it’s easier for the fraudsters to attack all the mid- and long-tail [digital service providers] as well, where they’re less likely to get caught and they’ll get a similar or better per-stream payout. Why not target all of these smaller DSPs with zero protections in place and get paid across all of them?
France’s CNM recently came to the conclusion that fraud is getting easier to commit.
BATEY: I 100% agree with that. There are so many ways to exploit platforms. If your job is to deliver the best user experience possible, it often means making it easy for them to access that content and creating really cool ways for them to experience or engage with that content. [When that happens,] there are more ways to manipulate that content for the purpose of exploiting it for a payout.
HAYDUK: And the tools that you need to commit fraud effectively and at scale are easier to access now than ever before. The tools that facilitate fraud in e-commerce or ticketing or financial services are also repackaged and repurposed to commit streaming fraud. You can generate fully automated online bot farms using cloud computing in a way you couldn’t 10 years ago.
How do you avoid generating false positives when you’re hunting for fraud?
HAYDUK: We know that a false positive is worth considerably more in the loss column than a false negative, so we adjust our models to account for the fact that they need to be conservative in the right ways.
BATEY: You can’t get it wrong. If you miss a fraudster, it’s OK. We hope we catch them later. If we call something fraud that’s not, that’s way worse.
Some have suggested that a user-centric payout system might mitigate fraud.
HAYDUK: Our view is that it’s not going to make that big a difference. It’ll change the tactics, but it won’t change the motivation. It’s a big pot of money on the internet, and generally speaking, the DSPs are still fairly soft targets. A different payout structure will just change the tactics that fraudsters use to aggregate money and divert it their way. Obviously, there’s a whole different case for the merits of payout systems if you’re an artist or you’re a label.
There’s a lot of industry concern about artificial intelligence right now. To what extent does AI make it even easier to commit these types of fraudulent activities?
HAYDUK: It’s a tool. We work for some good AI companies that care about not being a tool for fraudsters. That said, the new models are incredibly powerful, and you can create content at scale. There’s no putting the genie back in the bottle when some of these tools emerge. The tougher we make it to get away with fraud, the less valuable the tool becomes in the hands of someone who’s wielding it for a bad purpose.
How incentivized are DSPs to care about fraud?
HAYDUK: Their biggest partners care, especially in light of what we said earlier: Market share shifts matter to the partners and, therefore, it matters to the DSPs. I think consumers also care because bad recommendations on the DSP side make for bad user experience. And given that every platform is offering roughly the same catalog to the consumer, if your recommendations are substandard, that makes consumers more inclined to choose your competitor.
Some music industry executives worry that public discussions of fraud undermine user confidence.
HAYDUK: How many times a week does your bank email you about the extra efforts they’re taking to protect you from fraud in the financial sector? It doesn’t make me want to boycott my bank when they tell me that. Fans probably want to hear that, as an industry, we’re taking steps so that the artists they care about are paid correctly.
BATEY: If you’re the consumer, your account was hijacked, and now you’re getting a bunch of recommended songs that don’t make any sense, you’re not blaming the fraudster — you’re blaming the platform.
What is your dream scenario for fraud mitigation in the industry?
HAYDUK: Our view is there are some things you can’t do in a vacuum. DSP A can’t look at the data from DSP B to help inform its own detection models. It’s way too competitive between the platforms to give up the level of data required to do fraud detection at the highest levels. Having a platform in the middle acting as Switzerland, working for the collective benefit of everyone without minimizing the level of competitiveness between the platforms, is the right approach. And it’s also an approach that we’ve seen play out in other verticals with similar dynamics.