With the prior knowledge of video diffusion models, DiCoDe can compress a 2-second 16-frame video clip into 16 tokens. Despite the extremely high compression ratio, DiCoDe successfully reconstructs the video clips with minimal degradation.
Complex Scenes
Original
Reconstructed
Emerging Entities
Original
Reconstructed
Object Motion
Original
Reconstructed
Camera Motion
Original
Reconstructed
Videos are inherently temporal sequences by their very nature. In this work, we explore the potential of modeling videos in a chronological and scalable manner with autoregressive (AR) language models, inspired by their success in natural language processing. We introduce DiCoDe, a novel approach that leverages Diffusion-Compressed Deep Tokens to generate videos with a language model in an autoregressive manner.
Unlike existing methods that employ low-level representations with limited compression rates, DiCoDe utilizes deep tokens with a considerable compression rate (a 1000× reduction in token count). This significant compression is made possible by a tokenizer trained through leveraging the prior knowledge of video diffusion models.
Deep tokens enable DiCoDe to employ vanilla AR language models for video generation, akin to translating one visual "language" into another. By treating videos as temporal sequences, DiCoDe fully harnesses the capabilities of language models for autoregressive generation. DiCoDe is scalable using readily available AR architectures, and is capable of generating videos ranging from a few seconds to one minute using only 4 A100 GPUs for training.
We evaluate DiCoDe both quantitatively and qualitatively, demonstrating that it performs comparably to existing methods in terms of quality while ensuring efficient training. To showcase its scalability, we release a series of DiCoDe configurations with varying parameter sizes and observe a consistent improvement in performance as the model size increases from 100M to 3B.
We believe that DiCoDe's exploration in academia represents a promising initial step toward scalable video modeling with AR language models, paving the way for the development of larger and more powerful video generation models.
An animation of a hot air balloon.
A dramatic oil painting showcasing a stormy ocean.
A drone flying over a coastal town.
A time-lapse of clouds moving across a blue sky.
A time-lapse of a city skyline transitioning from day to night.
A bird taking a break on a sturdy fence post.
An elderly couple walking hand in hand, surrounded by a sunset's glow.
A black and white photograph of an old train traveling through the countryside.
A dramatic sunset over a calm sea.
A close-up of a chef's hands kneading dough.
A time-lapse of a sunset over a city skyline.
A boat along the river, with the Eiffel Tower in the back, style of Monet.
A solitary butterfly perched delicately on a blooming flower.
A lovely view of a lighthouse standing proud on a rocky shore.
An impressionist painting of a bustling market scene.
A magical underwater world where colorful fish dance gracefully.
A bustling city street filled with colorful storefronts.
A time-lapse of a busy city street during rush hour.
A solitary butterfly perched delicately on a blooming flower.
A playful kitten pouncing on a ball of yarn.
A close-up of a butterfly landing on a flower.
A single candle burning brightly in the dark.
A stunning oil painting depicting a stormy sea with waves crashing dramatically.
A time-lapse of clouds drifting across a blue sky.
A joyful girl dancing freely on the beach, moving in rhythm with the ocean's waves.
A digital animation of a robot exploring a futuristic city.
A drone flying over a winding river.
@misc{li2024dicodediffusioncompresseddeeptokens,
title={DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models},
author={Yizhuo Li and Yuying Ge and Yixiao Ge and Ping Luo and Ying Shan},
year={2024},
eprint={2412.04446},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.04446},
}