Passthrough camera access is generating a lot of buzz in the XR community lately. While we have a pretty good idea of where companies like Meta, Apple, and Pico stand, everyone’s curious about Google’s move with Android XR. I had the chance to chat directly with the team over at Google, and I can share that they’ll be rolling out a solution similar to what we see on phones. Let’s delve into the details!
The Camera Access Dilemma
For anyone unfamiliar, let’s take a moment to catch up. Most of today’s standalone VR headsets are actually Mixed Reality (MR) devices, displaying an RGB passthrough view captured by front cameras. This technology powers wonderful mixed reality creations like Cubism, Starship Home, and Pencil.
The operating system processes the images from these cameras to enable passthrough vision. As developers, we’re eager to tap into these camera feeds too. With access, we could use AI and computer vision to interpret the user’s environment and enhance the mixed reality experience significantly. In a past experiment, using a workaround on Quest, I managed to create an AI+MR prototype for home interior design, showcasing just how transformative this access could be.
This all sounds fantastic, yet there’s a significant hurdle: privacy concerns. If camera access fell into the wrong hands, malicious developers could capture surrounding images, run AI analyses, and potentially harvest sensitive information like bank details or ID numbers, not to mention misusing personal images.
It’s a complex issue. Balancing user privacy with the vast potential of mixed reality is crucial, but challenging.
How the XR Companies Are Responding
Initially, full camera access was available without restrictions. Some of you might recall my early experiments with camera textures on the Vive Focus back with my team at NTW around 2019. We explored diminished reality, Aruco marker tracking, and more.
As mixed reality gained traction, companies like Meta, Pico, HTC, and eventually Apple started restricting camera access due to privacy concerns. This practice became widespread, but the XR developer community began advocating for more openness, emphasizing the need for camera access. Folks like Cix Liv, Michael Gschwandtner, and I voiced that access should be transparent, ensuring users are fully aware when it’s being granted.
Our push led to breakthroughs, with Meta promising a “Passthrough API” early this year. Naturally, our curiosity shifted towards Google and Android XR.
Android XR: Embracing Phone-like Functionality
Android powers the majority of smartphones globally. On Android, apps can access camera streams through user permission. Once granted, developers can specify the camera ID—usually 0 for the back camera—and leverage the feed. Google aims to mirror this in Android XR, bringing the same seamless integration to its new operating system. After extensive conversations with a Google spokesperson, I can confirm this approach for Android XR:
Developers can utilize existing camera frames with user consent in XR, accessing streams through standard Android Camera APIs like Camera2 and CameraX. To access the main world-facing camera, akin to a smartphone’s back camera, apps must request appropriate permissions. Similarly, requesting the selfie-camera stream gains access to a user’s avatar feed, generated from tracking data via OpenXR APIs.
This setup means Android XR developers can employ familiar tools like CameraX to handle camera streams from XR headsets, enhancing functionality.
While the front-facing camera stream grants access, the rear camera involves a reconstructed avatar, mirroring Apple’s Vision Pro approach. This clever strategy maintains consistency between Android XR and traditional Android, managing expectations for both users and developers.
A Note on Android XR’s Future
Bear in mind, Android XR is still in the preview stage, and no commercial headset has released it yet. So, while the current landscape is promising, always stay prepared for adjustments post-final release.
Opening the Door to Camera Access
With Google and Meta pioneering this shift towards accessible camera feeds, it’s likely other industry players will adopt similar measures. By 2025, we could witness a major expansion in mixed reality innovation—an exciting prospect for developers worldwide!
(Header image is adapted from a Samsung image.)
Note: This blog contains advertisements and affiliate links to support its continuation. Clicking on an affiliate link earns a small commission. For more information, read my complete disclosure.
Consider sharing this with fellow innovators.
Related Posts
