FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

Zhikai Zhang1,3, Yitang Li1,3, Haofeng Huang1, Mingxian Lin3, Li Yi1,2,3,
1Tsinghua University, 2Shanghai AI Laboratory, 2Shanghai Qi Zhi Institute
Paper arXiv
Interpolate start reference image.

Abstract

Human motion synthesis is a fundamental task in computer animation. Despite recent progress in this field utilizing deep learning and motion capture data, existing methods are always limited to specific motion categories, environments, and styles. This poor generalizability can be partially attributed to the difficulty and expense of collecting large-scale and high-quality motion data. At the same time, foundation models trained with internet-scale image and text data have demonstrated surprising world knowledge and reasoning ability for various downstream tasks. Utilizing these foundation models may help with human motion synthesis, which some recent works have superficially explored. However, these methods didn't fully unveil the foundation models' potential for this task and only support several simple actions and environments. In this paper, we for the first time, without any motion data, explore open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment. Our framework can be split into two stages: 1) sequential keyframe generation by utilizing MLLMs as a keyframe designer and animator; 2) motion filling between keyframes through interpolation and motion tracking. Our method can achieve general human motion synthesis for many downstream tasks. The promising results demonstrate the worth of mocap-free human motion synthesis aided by MLLMs and pave the way for future research.

HumanAct12

Olympic sports

Style transfer

Stand+angry

Walk+old

HSI

Sit

Stepping stones

BibTeX

@misc{zhang2024freemotion,
      title={FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models}, 
      author={Zhikai Zhang and Yitang Li and Haofeng Huang and Mingxian Lin and Li Yi},
      year={2024},
      eprint={2406.10740},
      archivePrefix={arXiv},
      primaryClass={id='cs.CV' full_name='Computer Vision and Pattern Recognition' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers image processing, computer vision, pattern recognition, and scene understanding. Roughly includes material in ACM Subject Classes I.2.10, I.4, and I.5.'}
}