godot-docs/tutorials/xr/openxr_passthrough.rst

.. _doc_openxr_passthrough:

The OpenXR passthrough
======================

Passthrough is a technique where camera images are used to present the environment of the user as the background.
This turns a VR headset into an AR headset, often referred to as Mixed Reality or MR.

.. note::

  As passthrough is relatively new there isn't a singular way this is implemented across platforms.
  There may be additions in the future so this is a work in progress.

Passthrough extension
---------------------

OpenXR has a vendor extension for passthrough submitted by Meta.
Currently this extension is only supported on Quest but may be adopted by other headsets in the future.

:ref:`XRInterface <class_xrinterface>` has entry points for passthrough so different interfaces can implement this feature.
For :ref:`OpenXRInterface <class_openxrinterface>` the meta passthrough extension is implemented here.

In code you can call ``is_passthrough_supported`` to check if this extension is available.
If so you can simply enable passthrough by calling ``start_passthrough``.
You can call ``stop_passthrough`` to disable passthrough.

This will automatically set the main viewports ``transparent_bg`` property to true.
It will also result in the camera image being displayed as the background.
This will result in the background settings in the environment being ignored and alpha being applied.

.. note::

  For privacy reasons **no access** is given to the camera image.

.. warning::

  After passthrough is enabled it is possible to change settings that will break passthrough.
  Be sure not to change the ``transparent_bg`` setting or the environment blend mode.
  This will result in the camera image no longer being visible but you still incur the overhead.

  Always use ``stop_passthrough`` if you wish to turn off passthrough.

Finally, for using passthrough on the Quest you must set the following export property:

.. image:: img/xr_export_passthrough.webp

Passthrough through AR
----------------------

Some of the headsets recently adding OpenXR support have taken a different approach.
They simply mimic being an AR device. The Lynx R1 is such a device but others may be doing the same.

The following thus applies to both passthrough devices that mimic AR, and actual AR devices.

If ``is_passthrough_supported`` returns false the next step is to call ``get_supported_environment_blend_modes``.
This will return a list of supported blend modes for submitting the main render image to OpenXR.

We need to check if ``XR_ENV_BLEND_MODE_ALPHA_BLEND`` is present in this list.
If so we can tell OpenXR to expect an image that can be alpha blended with a background.
To do this, we simply call ``set_environment_blend_mode(xr_interface.XR_ENV_BLEND_MODE_ALPHA_BLEND)``.

We must also set ``transparent_bg`` to true to ensure we submit the right image.

Putting it together
-------------------

Putting the above together we can use the following code as a base:

.. code-block:: gdscript

  func enable_passthrough() -> bool:
    var xr_interface: XRInterface = XRServer.primary_interface
    if xr_interface and xr_interface.is_passthrough_supported():
      return xr_interface.start_passthrough()
    else:
      var modes = xr_interface.get_supported_environment_blend_modes()
      if xr_interface.XR_ENV_BLEND_MODE_ALPHA_BLEND in modes:
        xr_interface.set_environment_blend_mode(xr_interface.XR_ENV_BLEND_MODE_ALPHA_BLEND)
        return true
      else:
        return false