Skip to main content

Meta wants the virtual landscape to sound like real life

Meta and a group of researchers from the University of Texas at Austin (UT Austin) are working on bringing realistic audio to the metaverse.

As Kristen Garuman, Research Director at Meta AI, explains, there’s more to augmented and virtual reality (AR and VR, respectively) than just visuals. Audio plays a very important role in making a world feel alive. Garuman says “audio is shaped by the environment that [it’s] in.” There are various factors that influence how sound behaves like the geometry of a room, what’s in said room, and how far someone is from a source.

To achieve this, Meta's plan is to use AR glasses to record both audio and video from one location, then using a set of three AI models, transform and clean the recording so it feels like it's happening in front of you when you play it back at home. The AIs will take into account the room that you're in so it can match the environment. 

Looking at the projects, it appears Meta is focusing on AR glasses. Meta's plan for VR headsets includes replicating the sights and sounds of an environment, like a concert, so it feels like you're there in person.

We asked Meta how can people listen to the enhanced audio. Will people need a pair of headphones to listen or will it come from the headset? We didn't get a response.

We also asked Meta how can developers get a hold of these AI models. They've been made open source so third-party developers can work on the tech, but Meta didn't offer any further details.

Transformed by AI

The question is how can Meta record audio on a pair of AR glasses and have it reflect a new setting.

The first solution is known as AViTAR which is a ”Visual Acoustic Matching model.” This is the AI that transforms audio to match a new environment. Meta offers the example of a mother recording her child’s dance recital at an auditorium with a pair of AR glasses. 

One of the researchers claims that the mother in question can take that recording and play it back at home where the AI will morph the audio. It'll scan the environment, take into account any obstacles in a room, and have the recital sound like it's happening right in front of her with the same glasses. The researcher states the audio will come from the glasses.

To help clean up audio, there is Visually-Informed Dereverberation. Basically, it removes distracting reverb from the clip. The example given is recording a violin concert at a train station, taking it home, and having the AI clean up the clip so you hear nothing but music.

The last AI model is VisualVoice, which uses a combination of visual and audio cues to separate voices from other noises. Imagine recording a video of two people arguing. This AI will isolate one voice so you can understand them while silencing everything else. Meta explains visual cues are important because AI needs to see who’s talking in order to understand certain nuances and know who's speaking.

In relation to visuals, Meta states they plan on bringing in video and other cues to further enhance AI-driven audio. Since this technology is still early in development, it’s unknown if and when Meta will bring these AIs to a Quest headset near you.

Be sure to read our latest review on the Oculus Quest 2 if you're thinking of buying one. Spoiler alert: we like it.

Source: TechRadar

Popular posts from this blog

iPhone 13 Pro vs. iPhone 15 Pro Buyer's Guide: 50+ Differences Compared

The iPhone 15 Pro brings over 50 new features and improvements to Apple's high-end smartphones compared to the iPhone 13 Pro, which was released two years prior. This buyer's guide breaks down every major difference you should be aware of between the two generations and helps you to decide whether it's worth upgrading. The ‌iPhone 13‌ Pro debuted in 2021, introducing a brighter display with ProMotion technology for refresh rates up to 120Hz, the A15 Bionic chip, a telephoto camera with 3x optical zoom, Macro photography and photographic styles, Cinematic mode for recording videos with shallow depth of field, ProRes video recording, a 1TB storage option, and five hours of additional battery life. The ‌iPhone 13‌ Pro was discontinued upon the announcement of the iPhone 14 Pro in 2022, but it is still possible to get hold of it second-hand. Our guide helps to answer the question of how to decide which of these two iPhone models is best for you and serves as a way to c

The Best Early Black Friday iPhone Deals

We've begun tracking early Black Friday deals for major Apple products like AirPods , and now we're focusing on iPhone. Specifically, in this guide we'll share the best early Black Friday deals on iPhone 15, iPhone 15 Plus, iPhone 15 Pro, iPhone 15 Pro Max, and more. Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a purchase, we may receive a small payment, which helps us keep the site running. Of course, there is a chance that we could see better deals on iPhone arrive as Black Friday gets closer. This is a risk anyone has to take while shopping during the holidays, so with that in mind, many of the deals shared below are currently their best prices ever seen. More information on these sales can be found in our Black Friday Roundup . Best iPhone and Accessory Deals iPhone 15 Pro/Pro Max for up to $1,000 off at AT&T iPhone 15 Pro with free Apple TV 4K and Apple One six month subscription at Verizon AirTag

GiliSoft File Lock 3.4

A new program to encrypt and hide to protect files and folders and disks and make it safer. download :

Apple Store Taken Down Ahead of 'Scary Fast' Event Later Today

Apple has taken down its online store ahead of its "Scary Fast" event at 5 p.m. Pacific Time today . This is a common marketing tactic for the company. At the event, Apple is expected to announce an updated 24-inch iMac with the M3 chip, and updated 14-inch and 16-inch MacBook Pro models with M3 Pro and M3 Max chips. In addition, the Magic Keyboard, Magic Mouse, and Magic Trackpad for the Mac are expected to be updated with USB-C charging ports instead of Lightning. There is also a slim chance that a seventh-generation iPad mini will be announced at the event. The event is being held online, with no in-person component at Apple Park. A live stream will be available on Apple's website, on YouTube, and in the Apple TV app . Stay tuned for complete coverage of Apple's announcements. Tag: October 2023 Apple Event This article, " Apple Store Taken Down Ahead of 'Scary Fast' Event Later Today " first appeared on Discuss this arti