Tech Talk

The classic restaurant booth design gets a 21st century upgrade.

What you need to know

• Project Starline is a new way to remotely connect with people by providing the feeling of being present with them, known as telepresense.
• The booth uses an 8K display with a parallax barrier-like lens array to display 3D images without the use of glasses.
• User movement and audio are tracked via 3 "pods" that are used to render a life-like 3D version of a person to transmit wirelessly.

When Google announced Project Starline back at Google I/O 2021 in May, no one was under any illusion that the tech behind it was simple. A new whitepaper published by Google research, spotted by 9to5Google, details just how complicated the tech actually is.

It all starts with the booth design, which is similar to a familiar restaurant booth and gives users a comfortable place to sit while they meet or converse with other people remotely. Google says it chose a seated design primarily for this purpose.

Since comfort was paramount to the design, Google opted to develop a technology that didn't require users to wear anything — like any of the best VR headsets or 3D glasses — and still be able to see and hear the person on the other side as realistically as possible.

Google also wanted to ensure that users had the freedom to move around a bit, stretch their legs, and still get a full view of the person on the other end from head to waist.

Google built Project Starline booths with the ability to display a realistic 3D recreation of the person on the other end, complete with the ability to view them from multiple angles, maintain eye contact, and even hear audio as if it were coming from the other person's mouth.

In order to achieve this, Project Starline records and tracks individuals with a set of three "pods" — two located above the display on the left and right sides, and one centered below the display — which contains four synchronized 1280×1024 monochrome cameras operating at 120 Hz, with filters to block near-infrared light so that extra display information isn't erroneously picked up.

Think about how a Wii remote would freak out when sunlight poured into a room and you'll quickly understand the importance of blocking this type of light.

The bottom pod contains an extra RGB camera that captures the color necessary to properly replicate an individual in 3D space. Four cardioid microphones are also found throughout these pods, helping the system to replicate exactly where the sound comes from when it eventually recreates the person on the other end.

Visual and audio information captured from these cameras is then rendered on a Lenovo P920 PC with four GPUs — that's two NVIDIA Quadro RTX 6000 GPU and two NVIDIA GeForce Titan RTX GPUs — then sent over the internet via WebRTC. Essentially, a composite image is created using all of the pods' images combined into one big piece.

The 3D-rendered recreation of the person sitting in the Project Starline booth enables Google to not only send a higher quality image than can be delivered via 2D video as you'd find on Zoom, Skype, or FaceTime, but it also grants the ability to move and rotate the 3D model to create the illusion of a three-dimensional person sitting in front of you.

That display is a custom 65-inch 8K panel with stereoscopic lenses, which at least have a passing similarity to the display found in a Nintendo 3DS, which uses a parallax barrier. Unlike a parallax barrier, which has a small "sweet spot" where the 3D image is created, the lenses in Project Starline are used to send a different image to each eye, recreating 3D vision in a wholly natural way.

It can be argued that VR can create a similar experience in many ways, but Google points out that existing VR headsets don't have a wide enough field of view and have trouble properly recreating a face since a VR headset blocks quite a bit of it when worn.

Latency and other commonplace issues that still plague internet communication can also affect Starline a bit, but it seems like it might do a better job than what we've come to expect from current 2D video solutions.

Project Starline has been used by Google for many reasons, including on-boarding and meetings, but it's not clear how widely used we'll see Starline throughout offices or other commercial areas in the future. For now, check out Google's original announcement video below to see it in action.