Skip to main content

Meta wants the virtual landscape to sound like real life

Meta and a group of researchers from the University of Texas at Austin (UT Austin) are working on bringing realistic audio to the metaverse.

As Kristen Garuman, Research Director at Meta AI, explains, there’s more to augmented and virtual reality (AR and VR, respectively) than just visuals. Audio plays a very important role in making a world feel alive. Garuman says “audio is shaped by the environment that [it’s] in.” There are various factors that influence how sound behaves like the geometry of a room, what’s in said room, and how far someone is from a source.

To achieve this, Meta's plan is to use AR glasses to record both audio and video from one location, then using a set of three AI models, transform and clean the recording so it feels like it's happening in front of you when you play it back at home. The AIs will take into account the room that you're in so it can match the environment. 

Looking at the projects, it appears Meta is focusing on AR glasses. Meta's plan for VR headsets includes replicating the sights and sounds of an environment, like a concert, so it feels like you're there in person.

We asked Meta how can people listen to the enhanced audio. Will people need a pair of headphones to listen or will it come from the headset? We didn't get a response.

We also asked Meta how can developers get a hold of these AI models. They've been made open source so third-party developers can work on the tech, but Meta didn't offer any further details.

Transformed by AI

The question is how can Meta record audio on a pair of AR glasses and have it reflect a new setting.

The first solution is known as AViTAR which is a ”Visual Acoustic Matching model.” This is the AI that transforms audio to match a new environment. Meta offers the example of a mother recording her child’s dance recital at an auditorium with a pair of AR glasses. 

One of the researchers claims that the mother in question can take that recording and play it back at home where the AI will morph the audio. It'll scan the environment, take into account any obstacles in a room, and have the recital sound like it's happening right in front of her with the same glasses. The researcher states the audio will come from the glasses.

To help clean up audio, there is Visually-Informed Dereverberation. Basically, it removes distracting reverb from the clip. The example given is recording a violin concert at a train station, taking it home, and having the AI clean up the clip so you hear nothing but music.

The last AI model is VisualVoice, which uses a combination of visual and audio cues to separate voices from other noises. Imagine recording a video of two people arguing. This AI will isolate one voice so you can understand them while silencing everything else. Meta explains visual cues are important because AI needs to see who’s talking in order to understand certain nuances and know who's speaking.

In relation to visuals, Meta states they plan on bringing in video and other cues to further enhance AI-driven audio. Since this technology is still early in development, it’s unknown if and when Meta will bring these AIs to a Quest headset near you.

Be sure to read our latest review on the Oculus Quest 2 if you're thinking of buying one. Spoiler alert: we like it.

Source: TechRadar

Popular posts from this blog

Keep your Oculus Quest controllers going strong with these batteries

The Touch Controllers for the Oculus Quest 2 ship with one disposable AA battery each, but once those run out of juice, you should invest in the best Oculus Quest 2 replacement batteries to fill in for them. While the Touch Controllers last much longer than the headset's limited battery, it's still wise to invest in some rechargeable batteries or a stack of disposable batteries to stop your VR sessions from getting disrupted. Here are the batteries and chargers we recommend for your Oculus Touch controllers. Best rechargable batteries + charger Panasonic K-KJ55MCA4BA 3 Hour Quick Charger with 4 AA eneloop Rechargeable Batteries Staff Pick These rechargeable batteries store up to 2,000 mAh of power and can be recharged up to 2,100 times. They can be charged completely from dead or partially charged without damaging the energy storage memory. We recommend buying them with the quick charger accessory, which will get your AAs recharged in no time, but you can also purchas

Uber Eats exits seven markets, transfers one as part of competitive retooling

Uber Eats is pulling out of a clutch of markets — shuttering its on-demand food offering in the Czech Republic, Egypt, Honduras, Romania, Saudi Arabia, Uruguay and Ukraine. It’s also transferring its Uber Eats business operations in the United Arab Emirates (UAE) to Careem, its wholly owned ride-hailing subsidiary that’s mostly focused on the Middle East. “Consumers and restaurants using the Uber Eats app in the UAE will be transitioned to the Careem platform in the coming weeks, after which the Uber Eats app will no longer be available,” it writes in a regulatory filing detailing the operational shifts. “These decisions were made as part of the Company’s ongoing strategy to be in first or second position in all Eats markets by leaning into investment in some countries while exiting others,” the filing adds. An Uber spokesman said the changes are not related to the coronavirus pandemic but rather related to an ongoing “strategy of record” for the company to hold a first or s

This popular open-source web server has some serious security flaws

OpenLiteSpeed Web Server, a globally popular open-source web server , was carrying a couple of high-severity vulnerabilities, experts have warned. Threat actors that managed to exploit these flaws would have been given full privilege remote code execution capabilities, noted researchers from Unit 42, Palo Alto Networks’ cybersecurity research arm.  The team found OpenLiteSpeed Web Server carried three high-severity vulnerabilities, namely CVE-2022-0073 (an 8.8 severity score, high-severity remote code execution flaw), CVE-2022-0074 (an 8.8 high-severity privilege escalation flaw), and CVE-2022-0072 (a 5.8, medium-severity directory traversal flaw). The vulnerabilities also affected the enterprise version, LiteSpeed Web Server. Patch ready Unit 42 has notified LiteSpeed Technologies of its findings which has, subsequently, patched the flaws, and released new versions of the server, urging users to update their software immediately.  Organizations using OpenLiteSpeed versions 1.5.1

These Android games support Bluetooth controllers and they're better for it

Gaming is simply better with a controller in your hands. Gaming on Android typically requires you to settle for using touchscreen controls. However, some gracious game developers take the time and effort to add support for Bluetooth gamepads— and we love them for it. Since so few games provide this feature, we've taken the time to test and compiled our list of the best games that let you play with the best Bluetooth controller in your hands. We'd also recommend getting a Style Ring or PopSocket which can help prop your phone up at a good angle for gaming. You might recognize some of these games from the best Android games roundup and for good reason. But you're here for the best games with controller support, after all, so here I present to you the best you can find on the Play Store. The games Call of Duty Mobile GRID Autosport Tesla vs Lovecraft Evoland 2 Horizon Chase World Tour Riptide GP: Renegade Modern Combat 5: Blackout GTA: San Andreas Oceanhor