Overview

In this work, we enpower the LLM with multisensory perception and active observation abiltiy. We propose MultiPLY, a multisensory embodied large language model that could incorporate multisensory interactive data, including visual, audio, tactile, and thermal information into large language models, thereby establishing the correlation among words, actions, and perceptions. MultiPLY can perform a diverse set of multisensory embodied tasks, including multisensory question answering, embodied question answering, task decomposition, object retrieval, and tool use.

How to enpower LLMs with embodied multisensory perception ability?

Demos

(enable audio in the control bar to hear the sound)

@misc{hong2024multiply,
      title={MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World}, 
      author={Yining Hong and Zishuo Zheng and Peihao Chen and Yian Wang and Junyan Li and Chuang Gan},
      year={2024},
      eprint={2401.08577},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}