Bailando++: 3D Dance GPT With Choreographic Memory
In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Jg. 45 (2023-12-01), Heft 12, S. 14192-14207
Online
serialPeriodical
Zugriff:
Our proposed music-to-dance framework, Bailando++, addresses the challenges of driving 3D characters to dance in a way that follows the constraints of choreography norms and maintains temporal coherency with different music genres. Bailando++ consists of two components: a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequences, and an actor-critic Generative Pre-trained Transformer (GPT) that composes these units into a fluent dance coherent to the music. In particular, to synchronize the diverse motion tempos and music beats, we introduce an actor-critic-based reinforcement learning scheme to the GPT with a novel beat-align reward function. Additionally, we consider learning human dance poses in the rotation domain to avoid body distortions incompatible with human morphology, and introduce a musical contextual encoding to allow the motion GPT to grasp longer-term patterns of music. Our experiments on the standard benchmark show that Bailando++ achieves state-of-the-art performance both qualitatively and quantitatively, with the added benefit of the unsupervised discovery of human-interpretable dancing-style poses in the choreographic memory.
Titel: |
Bailando++: 3D Dance GPT With Choreographic Memory
|
---|---|
Autor/in / Beteiligte Person: | Siyao, Li ; Yu, Weijiang ; Gu, Tianpei ; Lin, Chunze ; Wang, Quan ; Qian, Chen ; Loy, Chen Change ; Liu, Ziwei |
Link: | |
Zeitschrift: | IEEE Transactions on Pattern Analysis and Machine Intelligence, Jg. 45 (2023-12-01), Heft 12, S. 14192-14207 |
Veröffentlichung: | 2023 |
Medientyp: | serialPeriodical |
ISSN: | 0162-8828 (print) |
DOI: | 10.1109/TPAMI.2023.3319435 |
Sonstiges: |
|