Although 360° cameras ease the capture of panoramic footage, it remains challenging to add realistic 360° audio that blends into the captured scene and is synchronized with the camera motion. We present a method for adding scene-aware spatial audio to 360° videos in typical indoor scenes, using only a conventional mono-channel microphone and a speaker. We observe that the late reverberation of a room's impulse response is usually diffuse spatially and directionally. Exploiting this fact, we propose a method that synthesizes the directional impulse response between any source and listening locations by combining a synthesized early reverberation part and a measured late reverberation tail. The early reverberation is simulated using a geometric acoustic simulation and then enhanced using a frequency modulation method to capture room resonances. The late reverberation is extracted from a recorded impulse response, with a carefully chosen time duration that separates out the late reverberation from the early reverberation. In our validations, we show that our synthesized spatial audio matches closely with recordings using ambisonic microphones. Lastly, we demonstrate the strength of our method in several applications.

---

materials

Paper / Paper (low resolution) / arxiv
Youtube / Video (100MB)
Slides: keynote (350MB) / pdf (30MB)

Hardware: Ricoh Theta V 360 Camera / TA-1 3D Audio Microphone / Zoom H2n Recorder / Presonus Eris E3.5 Reference Speaker

Data: SpEAR speech database

---

slides quickview

---

acknowledgements

We thank Chunxiao Cao for discussing and sharing his bidirectional sound simulation code, Zhili Chen for sharing the SfM code, Carl Schissler for sharing the "infinite" audio file, James Traer for discussion on IR measurement, and Henrique Maia for proofreading and voiceover. This work was supported in part by the National Science Foundation (CAREER-1453101), SoftBank Group, and generous gift from Adobe. Dingzeyu Li was partially supported by an Adobe Research Fellowship.

---

bibtex citation

@article{Li:2018:360audio,
  title={Scene-Aware Audio for 360\textdegree{} Videos},
  author={Li, Dingzeyu and Langlois, Timothy R. and Zheng, Changxi},
  journal={ACM Trans. Graph.},
  volume={37},
  number={4},
  year={2018},
}
  

---