Vision AI Co-Processing for Dynamic Context Awareness in Mixed Reality

[2024 – 2025]

Real-time offloading of semantic perception tasks from MR headsets to edge AI accelerators, enabling scene-level understanding and interaction in unstructured environments.

This research investigates a co-processing framework for real-time visual inference in mixed reality (MR), targeting the computational limitations of current head-mounted displays. While modern MR devices support spatial tracking, they lack the capacity to perform high-resolution semantic scene analysis onboard. To address this, we propose an architecture in which semantic segmentation, object detection, and classification are offloaded to external edge AI accelerators.

The system integrates commercial MR headsets (e.g. Magic Leap 2, Meta Quest) with compact processing nodes equipped with devices such as the Hailo-8. Visual input—either full-resolution frames or compressed intermediate features—is transmitted over low-latency wireless links. Inference is performed externally, and structured outputs (e.g. object masks, bounding boxes, scene labels) are returned to the headset and rendered in-engine via Unity or Unreal.

This enables applications to interact meaningfully with their physical surroundings. Virtual content can be dynamically anchored to recognized objects, navigation systems can adapt to context-dependent spatial cues, and application logic can respond to real-time environmental classification. Unlike traditional onboard pipelines, this approach decouples perception from headset constraints, allowing for broader model complexity and higher update rates.

Initial deployments include mobile outdoor scenarios with sub-100 ms round-trip latency, as well as indoor annotation tasks requiring dense semantic feedback. Multi-headset use cases have also been validated through shared inference nodes.

By enabling structured, real-time perception at the scene level, this research contributes to the advancement of Spatial AI: the integration of geometric mapping, semantic interpretation, and real-world interactivity within mixed reality systems.

Publication

• Context-Rich Interactions in Mixed Reality through Edge AI Co-Processing
  Orsholits, Alex, Tsukada, Manabu. The 39-th International Conference on Advanced Information Networking and Applications (AINA 2025), Barcelona, Spain, 2025, ISBN: 978-3-031-87771-1 [Proceedings Article]BibTeX
• Edge Vision AI Co-Processing for Dynamic Context Awareness in Mixed Reality
  Orsholits, Alex, Tsukada, Manabu. IEEE VR 2025, Poster, 2025, (Honorable mention) [Miscellaneous]BibTeX
• A Distributed Content Subscription Mechanism with Revision Discovery to Decouple Content Sharing Platform and Creator ID
  Zhu, Zhihai, Tao, Ye, Tsukada, Manabu, Esaki, Hiroshi. International Conference on Artificial Intelligence in Information and Communication (ICAIIC 2025) , Fukuoka, Japan, 2025 [Proceedings Article]BibTeX