InterFusion: Text-Driven Generation of 3D Human-Object Interaction

Sisi Dai1, Wenhao Li1, Haowen Sun2, Haibin Huang3, Chongyang Ma3, Hui Huang2, Kai Xu1* Ruizhen Hu2*
1National University of Defense Technology, 2Shenzhen University, 3Kuaishou Technology
*Corresponding Authors

ECCV 2024
Paper arXiv Code

Abstract

In this study, we tackle the complex task of generating 3D human-object interactions (HOI) from textual descriptions in a zero-shot text-to-3D manner. We identify and address two key challenges: the unsatisfactory outcomes of direct text-to-3D methods in HOI, largely due to the lack of paired text-interaction data, and the inherent difficulties in simultaneously generating multiple concepts with complex spatial relationships. To effectively address these issues, we present InterFusion, a two-stage framework specifically designed for HOI generation. InterFusion involves human pose estimations derived from text as geometric priors, which simplifies the text-to-3D conversion process and introduces additional constraints for accurate object generation. At the first stage, InterFusion extracts 3D human poses from a synthesized image dataset depicting a wide range of interactions, subsequently mapping these poses to interaction descriptions. The second stage of InterFusion capitalizes on the latest developments in text-to-3D generation, enabling the production of realistic and high-quality 3D HOI scenes. This is achieved through a local-global optimization process, where the generation of human body and object is optimized separately, and jointly refined with a global optimization of the entire scene, ensuring a seamless and contextually coherent integration. Our experimental results affirm that InterFusion significantly outperforms existing state-of-the-art methods in 3D HOI generation.

Keywords: Text-Driven Generation · Zero-Shot Generation · 3D Human-Object Interaction Generation

Method Overview


InterFusion is a two-stage framework that transforms textual descriptions into detailed 3D human-object interactions, initially synthesizing anchor poses and then optimizing human and object models with constraints from estimated pose and textual prompts.

Visual Results

BibTeX

@inproceedings{dai2024interfusion,
      title={InterFusion: Text-Driven Generation of 3D Human-Object Interaction},
      author={Dai, Sisi and Li, Wenhao and Sun, Haowen and Huang, Haibin and Ma, Chongyang and Huang, Hui and Xu, Kai and Hu, Ruizhen},
      booktitle={ECCV},
      year={2024}
    }