Song-Adaptive Music Visualization

Song-Adaptive Music Visualization is my diploma thesis in Visual Computing at TU Wien (in progress), advised by Univ.Prof. Michael Wimmer with co-supervision from Stefan Ohrhallinger. The project asks whether Large Language Models can help music visualizations adapt to a song’s larger structure—verses, choruses, mood, and instrumentation—rather than only reacting to volume, beats, or frequency bands.

The approach is a hybrid pipeline that keeps creative mapping and real-time graphics separate. Offline, a Python/FastAPI backend runs Music Information Retrieval: Demucs stem separation, global tempo and key estimation, local rhythmic/melodic/timbral features, and section segmentation into descriptors. A multi-stage LLM prompt chain then turns those descriptors into a validated configuration.json—aesthetic interpretation, direction planning, then schema-constrained compilation with Pydantic validation and retries.

At runtime, a deterministic WebGPU frontend applies that configuration in sync with playback. Precomputed features drive shader uniforms through a UniformBridge with smoothing and fixed slot mapping; pipelines are prewarmed so section changes remaps uniforms and layer opacities instead of recompiling shaders. Evaluation will combine user ratings of audio-visual alignment and section awareness with quantitative checks of timing correspondence and frame-time stability, compared against simpler baseline reactive visualizers.