Skip to main content

Meta wants the virtual landscape to sound like real life

Meta and a group of researchers from the University of Texas at Austin (UT Austin) are working on bringing realistic audio to the metaverse.

As Kristen Garuman, Research Director at Meta AI, explains, there’s more to augmented and virtual reality (AR and VR, respectively) than just visuals. Audio plays a very important role in making a world feel alive. Garuman says “audio is shaped by the environment that [it’s] in.” There are various factors that influence how sound behaves like the geometry of a room, what’s in said room, and how far someone is from a source.

To achieve this, Meta's plan is to use AR glasses to record both audio and video from one location, then using a set of three AI models, transform and clean the recording so it feels like it's happening in front of you when you play it back at home. The AIs will take into account the room that you're in so it can match the environment. 

Looking at the projects, it appears Meta is focusing on AR glasses. Meta's plan for VR headsets includes replicating the sights and sounds of an environment, like a concert, so it feels like you're there in person.

We asked Meta how can people listen to the enhanced audio. Will people need a pair of headphones to listen or will it come from the headset? We didn't get a response.

We also asked Meta how can developers get a hold of these AI models. They've been made open source so third-party developers can work on the tech, but Meta didn't offer any further details.

Transformed by AI

The question is how can Meta record audio on a pair of AR glasses and have it reflect a new setting.

The first solution is known as AViTAR which is a ”Visual Acoustic Matching model.” This is the AI that transforms audio to match a new environment. Meta offers the example of a mother recording her child’s dance recital at an auditorium with a pair of AR glasses. 

One of the researchers claims that the mother in question can take that recording and play it back at home where the AI will morph the audio. It'll scan the environment, take into account any obstacles in a room, and have the recital sound like it's happening right in front of her with the same glasses. The researcher states the audio will come from the glasses.

To help clean up audio, there is Visually-Informed Dereverberation. Basically, it removes distracting reverb from the clip. The example given is recording a violin concert at a train station, taking it home, and having the AI clean up the clip so you hear nothing but music.

The last AI model is VisualVoice, which uses a combination of visual and audio cues to separate voices from other noises. Imagine recording a video of two people arguing. This AI will isolate one voice so you can understand them while silencing everything else. Meta explains visual cues are important because AI needs to see who’s talking in order to understand certain nuances and know who's speaking.

In relation to visuals, Meta states they plan on bringing in video and other cues to further enhance AI-driven audio. Since this technology is still early in development, it’s unknown if and when Meta will bring these AIs to a Quest headset near you.

Be sure to read our latest review on the Oculus Quest 2 if you're thinking of buying one. Spoiler alert: we like it.

Source: TechRadar

Popular posts from this blog

Follow these steps to connect a Pro Controller to your Android phone

Playing games on your smartphone is one of the best ways to entertain yourself. However, it can be tough to play with some games when you're just tapping on a screen. Fortunately, it's possible to sync up a traditional controller. That's where it's nice to connect your Nintendo Switch Pro Controller and get playing on the best gaming phones . By the way, the Playstation 4 controller as well as the Xbox One controller are also compatible with Android devices, if you'd prefer to use one of those. Note: You will only be able to use a Pro Controller if your phone is running Android 10 and if the game you're playing supports controllers. Additionally, the process for syncing the controller with your phone will be different from one phone to the next. How to use Switch controller on Android: Sync Pro Controller to your phone via Bluetooth Do keep in mind that some Android games — including some of the most popular titles like Genshin Impact — don't act

FCC approves broadband 'nutrition labels' to help you shop for internet

The FCC is pushing nutrition labels for internet providers. What you need to know The FCC has voted to move forward with new rules for ISPs to display nutrition labels. The proposed rulemaking would mandate ISPs to display relevant speed and pricing information to consumers. This should make it easier for consumers to make an informed decision on their broadband. The FCC voted unanimously on a plan that would allow consumers to make better decisions about their broadband internet. The proposal will require internet service providers (ISPs) - including many of the best wireless carriers in the U.S. — to display "nutrition labels" that display relevant service information for consumers at point-of-sale. This includes internet speeds, allowances, and clear information on rates. "If you walk into any grocery store and pull boxes of cereal from the shelves, you can easily compare calories and carbohydrates," FCC Chair Jessica Rosenworcel said in a statemen

I replaced my Steam Deck’s noisy fan and am so happy I did

Photo by Sean Hollister / The Verge 160 hours into Elden Ring, I’m sure of one thing: the single most annoying thing about my Steam Deck is its whiny fan. And now, I’m so happy to report there’s a way to fix it. It takes roughly 15 minutes and $30 — depending on where you live — to install a replacement fan from iFixit. Five days ago, the repair company finally got a large shipment of those fans , and I bought one right away. Now, my Steam Deck’s tiny screech is gone. Is the fan still loud? Yes, yes it is, but it’s a whoosh instead of a whine. I like to think of it as the sound of air escaping the Deck’s vents, but mostly, I don’t think about it at all. I can easily tune out the new whoosh, whereas the whine always managed to get my attention even after Valve tweaked its software to bring the fan noise down. In fact, the new fan has less whine than the electrical tape trick I showed you in April , and — YMMV — seems to be oh-so-slightly quieter overall. I did an admittedly

You can make your new Pixel look like a Teenage Mutant Ninja Turtle

A cool thing about Google’s Pixel 6 series phones is their unique camera bar design . If the finish of that strip were a different color than black — specifically blue, orange, purple, or red — it would totally look kind of like the Teenage Mutant Ninja Turtles . And now that you can’t unsee the resemblance, device outfitter Dbrand would like to help make your Pixel 6 device look like one of your favorite childhood heroes. With Google’s latest phone the Pixel 6a releasing next week , Dbrand decided it's a good time to revisit the sewers and launch its Teenage Mutant Ninja Pixels decals for all three phones. They match the green heroes-in-a-half-shell plus their masks, and include four camera decal strips that reflect the colors of Leonardo, Michelangelo, Donatello, and Raphael. Personally, I think the Pixel 6a’s two-camera array does the best job looking like proper eyes (the spidery camera array of the Pixel 6 and 6 Pro, not so much). Available now for the Pixel 6a: https://