Design of Digital Twin-Based Infrastructure for 3D Audio Spatialization

[2021 – 2025]

Building a digital twin platform that fuses large-scale geometry, precise positioning, and real-time acoustic simulation to enable responsive 3D audio in urban mixed-reality environments.

As interest in immersive spatial experiences continues to grow, Tsukada Lab investigates how digital‑twin technology can be leveraged to enable accurate, real‑time 3D audio environments for mixed‑reality applications. This research focuses on cloud-assisted acoustic simulation informed by either large-scale urban geometric models or BIM data, listener positioning, and network-based rendering infrastructure. The goal is to render dynamic spatial audio that behaves realistically with respect to complex geometries and listener movement.

The platform architecture combines large-scale 3D city models or BIM data with RTK-GNSS or SLAM-based tracking systems to determine the listener’s precise location and orientation. This data is fed into spatialization engines that simulate acoustics in real time. Leveraging parallelized server compute for wave-based audio propagation, the system generates soundscapes that reflect environmental features such as occlusion, diffraction, and reverberation. The listener experiences a rich and reactive audio environment that evolves with their movement in physical space.

Several prototypes and field deployments have demonstrated the feasibility of this approach. These include mobile audio visualization systems using smartphones and AR frameworks, remote-rendered immersive urban soundscapes, and real-time synchronization with building-scale acoustic simulations. These implementations validate the technical architecture while providing data to assess simulation accuracy across different environments.

By integrating geospatial infrastructure, embedded sensing, and audio rendering pipelines, the project advances the state of the art in spatial computing and urban informatics. It lays the foundation for a programmable audio infrastructure that can enhance urban navigation, cultural experiences, and situational awareness through immersive, context-aware sound design.

Publication

• PLATONE: Assessing Simulation Accuracy of Environment-Dependent Audio Spatialization
  Orsholits, Alex, Nardini, Eric, Manabu, Tsukada. International Conference on Intelligent Computing and its Emerging Applications (ICEA2024), pp. 57 - 62, Tokyo, Japan, 2024 [Proceedings Article]BibTeX
• PLATONE: An Immersive Geospatial Audio Spatialization Platform
  Orsholits, Alex, Qian, Yiyuan, Nardini, Eric, Obuchi, Yusuke, Tsukada, Manabu. The 2nd Annual IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom 2024), Hong Kong, China, 2024 [Proceedings Article]BibTeX
• Design of Digital Twin Architecture for 3D Audio Visualization in AR
  Takada, Tokio, Nakazato, Jin, Orsholits, Alex, Tsukada, Manabu, Ochiai, Hideya, Esaki, Hiroshi. The 2nd Annual IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom 2024), Hong Kong, China, 2024 [Proceedings Article]BibTeX