Figureure 3 · : Method illustrationFig. 3: Method illustration. Cubes , squares <sup>□</sup>, and circles respectively represent input patches or patch-wise tokens, holistic discrete tokens, and continuous query embeddings. (a) AVTok features a dual-stream transformer-based architecture, of which each stream’s forward pass is demonstrated in (b), to jointly learns video (Blue stream) and audio (Green stream) reconstructions in a unified holistic scheme. It leverages separate sets of learnable queries and normalization layers to gather modal-specific information, while sharing remaining parameters to enable implicit cross-modal interaction, achieving both eficiency and eficacy. In addition to the standard reconstruction training objectives $\mathcal { L } _ { r e ( } ^ { v }$ and $\mathcal { L } _ { r e c } ^ { a } ,$ we align AVTok’s patch-wise continuous tokens with an audio-visual foundation model $\mathcal { M } _ { F }$ via $\mathcal { L } _ { \boldsymbol { r } \boldsymbol { e p } }$ to better capture synergistic features between auditory and visual elements. Lastly, an AR prior model $\mathcal { M } _ { P }$ is also equipped to encourage an AR-friendly discrete latent space via $\mathcal { L } _ { p r i o r : }$ , facilitating downstream AR generative tasks including (c) audio-to-video, (d) video-to-audio, and (e) class-conditional joint audio-video generation.这张图概括 AVTok 的整体方法流程。阅读时先看模块之间传递的训练信号，再看作者如何把目标拆成可优化的子问题。Figureure 1 · : HighlightsFig. 1: Highlights. (a) We propose AVTok, a novel unified tokenizer with dual-stream transformer-based architecture, capable of jointly encoding an audio-video pair into a single compact 1D latent representation. (b) AVTok achieves competitive performance compared to state-of-the-art unimodal 1D video tokenizers (top) and audio codecs (bottom). (c) From left to right, AVTok can be integrated into AR generative models to achieve audio-to-video, video-to-audio, and joint audio-video generation.这张图概括 AVTok 的整体方法流程。阅读时先看模块之间传递的训练信号，再看作者如何把目标拆成可优化的子问题。
