The whole thesis fits in this diagram. A prompt goes through the Gemma-3 text encoder once, then the dual-stream diffusion transformer, with a 14 billion parameter video stream and a 5 billion parameter audio stream, refines noise for many steps, sometimes three calls per step. The decoders run once at the end. Read the cost strip at the bottom: the indigo box is both the largest and the most repeated part, so that is what gets compressed first.

Memory is simply parameters times bytes per parameter. The teal bars are 16-bit weights, the pale rust bars 8-bit, the solid rust bars 4-bit. The transformer drops from 38 gigabytes to about 10, and Gemma from 24 to about 6. Smaller weights also mean fewer bytes to move at every step, which is where the speed comes from on memory-limited hardware. The thesis results table will pair each of these settings with latency and quality.

Diffusion builds a video by removing noise a little at a time. Read left to right: pure noise at timestep one, a clean latent at timestep zero, and every indigo arrow is one full call of the transformer. The gold axis is the timestep, the noise level. Because the same network sees very noisy inputs early and nearly clean ones late, its activations change over time, which is why the diffusion quantization papers keep talking about timesteps.

This is one block of a diffusion transformer, and the model stacks many of them. Tokens pass self-attention, cross-attention to the text, and a feed-forward network, each with a residual shortcut. The gold path shows AdaLN: a small network turns the timestep into scale, shift and gate values that modulate each block. In LTX-2 the dashed input adds cross-attention between the audio and video streams. Nearly every box is made of linear layers.

Each row is one number format, drawn to scale, one cell per bit. Floating-point formats split bits into sign, exponent for range, and mantissa for detail; BF16 keeps FP32's range with less detail, FP16 the reverse. INT8 and INT4, in rust, are plain evenly spaced levels, 256 or 16 of them, so they need a separate scale to mean a real value. Bytes per parameter on the right drive the memory table below.

A worked example of equations four to six. The top line is real values, the rust ticks below are the sixteen INT4 levels. With a range from minus 0.6 to 0.9, the step is 0.1 and zero sits at level 6. The value 0.23 lands on level 8 and comes back as 0.2, an error of 0.03. The magenta 1.3 lies outside the range, is clipped to level 15, and returns as 0.9, an error of 0.4.

Granularity means which numbers share a scale. From left to right: one scale for the whole matrix, one per row or output channel, one per group of inputs such as 128, and for activations one per token computed at run time, in gold. Darker shades stand for different scale values. Finer sharing fits the data better but stores more scales. NNCF typically uses per-channel scales for INT8 and group-wise scales for INT4.

Two families of quantization. On the left, weight-only W four A sixteen: INT4 weights are expanded to 16-bit inside the kernel and multiplied with untouched 16-bit activations, saving memory and bandwidth. On the right, W8A8: activations are also quantized so the multiply runs in integer arithmetic, which needs matching hardware and makes activation outliers dangerous. The core track of the thesis is the left side, with W8A8 as an optional comparison.

The chart on the left shows the typical picture in large transformers: most channels have small values, and a few magenta channels are huge. The bars are illustrative. On the right, a single shared scale must stretch to cover the outlier, so ordinary values squeeze into one or two levels and lose almost all detail. The fixes listed at the bottom are the main ideas of SmoothQuant, AWQ, ViDiT-Q and SVDQuant.

Three terms that appear in every paper. Post-training quantization, or PTQ, needs only the trained model and a small calibration set, which for a diffusion model must cover many timesteps. Quantization-aware training, or QAT, retrains with rounding simulated and costs days of GPU time, like QVGen. The bottom strip shows mixed precision: most layers in INT4, a few sensitive ones in INT8 or 16-bit. The pattern is illustrative.

Two kinds of chips and the memory ladder behind them. A CPU has a few strong cores with vector units and large caches; a GPU has many small cores with matrix units and its own video memory. The pyramid on the right shows the rule: the closer to the cores, the faster and smaller. LTX-2.3 is far too big for any cache, so every step must stream its weights from main memory.

Many papers report speed from custom CUDA kernels on NVIDIA GPUs such as the A100, H100 or RTX 4090, using Tensor Cores for their low-bit format. The thesis stack on the right uses only OpenVINO's built-in kernels, CPU vector units and the XMX matrix engines of Arc. The magenta gaps are the reasons a speed-up may not carry over. Quality findings usually transfer, but every speed claim has to be measured again.

This is the actual code path of the project. The diffusers model in PyTorch is exported by Optimum Intel into OpenVINO IR, an xml graph and a bin weight file per component. NNCF then compresses the weights to INT8 or INT4, with options such as AWQ, scale estimation and GPTQ. The compressed IR is compiled for CPU or GPU and run by the OpenVINO pipeline. The gold harness compares every result with the PyTorch reference.

Metrics answer different questions. The teal box holds paired metrics that compare with the full-precision output for the same prompt and seed, including the WWB similarity used in this project. The white box holds unpaired metrics that judge realism, text match or video dimensions, like FID, CLIP score and VBench. The lower boxes cover language models, audio and human judgement. The paper names show who uses what, so numbers can be compared correctly.

Each row is a timeline of denoising steps, and shorter means faster. Quantization shortens every step. Step caching reuses results when the gold signal says little has changed, which is what TeaCache does. Distillation cuts the number of steps but needs training, so it is outside the scope of the thesis. The last row shows that quantization and caching combine, with gains multiplying and errors adding up. The bar lengths are illustrative.

This map groups the sixteen papers into five columns. Indigo boxes are the ones the thesis uses directly: LTX-2 as the target, GPTQ and AWQ inside NNCF, ViDiT-Q's sensitivity method, and VBench. Dashed indigo boxes, SmoothQuant and TeaCache, are optional tracks. Arrows show which ideas are reused, for example SmoothQuant scaling in PTQ4DiT, ViDiT-Q and SVDQuant, and SVDQuant plus GPTQ in the Wan 2.2 paper.

The whole paper in one picture: the top row is a typical video latent diffusion pipeline, the bottom row is LTX-Video. LTX moves patchifying into the VAE and hands the final denoising step to the decoder. For 121 frames at 768 by 512 that cuts the token count from about 47 thousand to about 6 thousand, 7.75 times fewer, and since attention grows with tokens squared, roughly 60 times less attention work. Every later block explains one piece of this diagram.

Sample outputs, drawn schematically since the originals are photos. The first row is text-to-video from a long prompt, and the next two rows are image-to-video, where the gold frame on the left is the given image and the teal frames are generated over five seconds. The point is prompt adherence and coherent motion. These are the qualities the compressed model must preserve.

The holistic denoising idea as a flow. Start with a pure-noise latent of size h over 32, w over 32, and f plus 7 over 8. The transformer denoises it down to noise level t two, then the decoder takes the latent at t one and outputs the clean full-size video. This is the paper's Figure 2 and the visual form of equations u5 and u6.

Training makes the 128 latent channels independent, and so dense. Panel a plots cumulative explained variance against principal component: early curves in blue rise fast, meaning a few directions hold everything, while the final curve in rust is almost a straight line, meaning all channels matter. Panels b and c show channel correlations fading from strong to near zero. For the thesis, a dense latent means little redundancy to hide quantization error.

The Video-VAE architecture. Panel a is the causal encoder: patchify, then stacks of residual blocks with three downsampling stages, giving a 128-channel latent at 32 by 32 by 8. Panel b is the denoising decoder, whose conditioned residual blocks inject the timestep by scale and shift and add learned noise. The exact patchify split is the tab's inference. The decoder is conv-heavy, which matters when deciding what to quantize.

The reconstruction GAN in two panels. In panel a, a normal discriminator sees one image and guesses real or fake. In panel b, it sees the original and its reconstruction side by side, in either order, and must say which is the original. This relative task is easier and more stable, and it also pushes reconstructions closer to the input. It is the rust loss in Method 1.

One of the 28 transformer blocks. Latent tokens pass through self-attention with rotary positions and QK normalisation, then cross-attention to T5 text tokens, then a feed-forward network, each with a residual. The timestep enters in gold through AdaLN scale and shift, and in LTX it can differ per token. These attention and feed-forward matrix multiplies are the layers NNCF weight compression will target.

Three ways to give tokens positions for rotary embeddings. Panel a uses integer indices, panel b uses fractions of the current size, and panel c, which works best, uses fractions of a fixed maximum size, measured in pixels and seconds. With c, the same place keeps the same code across resolutions, lengths and frame rates. This is part of the transformer design in the method steps.

Two heat maps of rotary frequencies, each cell being position times frequency. On the left, LTX's exponential spacing covers a useful range of rotation speeds. On the right, the common inverse-exponential spacing leaves most channels barely rotating. The colours are redrawn, not exact. Figure 17 shows the payoff in training loss.

How image-to-video works without extra parameters. The input frame is encoded into one latent frame and placed before the noise tokens. Its tokens get a small timestep t c, often zero, meaning clean, while all other tokens get t equal to 1. Each token keeps its 3-D position. This per-token timestep is why activation ranges vary across tokens in the same step.

Training noise levels shift toward high noise as token count grows. Each panel plots the probability of sampling each t: the left for about 256 tokens, the right for about 4096. The shifted curve peaks at higher t, and the teal stretched version that LTX uses is clamped so very low and very high t are still sampled. The shapes are simulated, not copied from the paper.

The data pipeline, read left to right and then back. Raw shots are cropped, static clips are removed by a motion check, a CLIP embedding and an aesthetic score are computed, then clips are clustered, de-duplicated, filtered and resized into the final training set. It explains where quality comes from, but it is not relevant to compression.

Examples of the internal captioner. Each row is a video, here generated by Sora, with the long structured caption it received: subject, action, setting, camera motion and lighting, ending with the style. The whole training set was re-captioned this way. It explains why LTX works best with long, detailed prompts, which is worth copying in the evaluation prompts.

A word cloud of the training captions, with word size roughly showing frequency. Colours, people, camera and background terms dominate, along with the phrase real-life footage. The sizes are approximate. It shows what kind of prompts the model has seen most, which helps in picking realistic test prompts.

Two histograms of the training data. Panel a shows words per caption, peaking around 75 to 85 words with a long tail. Panel b shows clip length, mostly under five seconds with a tail to thirty. The bar heights are read off the original plot. Together they say the model is trained on long captions and short clips.

Pairwise human preference matrices, text-to-video in panel a and image-to-video in panel b. Each cell is how often the row model beat the column model, and darker is higher. The bottom row, LTX, wins between 72 and 96 percent of its matchups. This is the detailed version of Table 2, and the baseline the compressed model's quality should be compared against.

A hard reconstruction case, drawn schematically: a lighthouse covered in scaffolding, full of thin high-frequency lines. With a standard GAN loss the scaffold smears into artifacts, while with the reconstruction GAN it stays close to the input. This is the evidence for the rust loss in Method 1, and a reminder that fine detail is fragile at 1 to 192 compression.

Training loss against step for two otherwise identical runs. The rust curve uses inverse-exponential rotary frequencies and stays above the indigo exponential curve for all 140 thousand steps, ending around 0.59 versus 0.58. The gap is small but consistent. It justifies the frequency choice shown in Figure 8.

More text-to-video samples, drawn as a schematic grid. Each row is one prompt, and the frames are evenly spaced over five seconds. In the paper, the prompt words the video realises are coloured green, and most of each prompt is green. The takeaway is prompt adherence on detailed cinematic prompts.

More image-to-video samples, drawn schematically. The gold frame on the left of each row is the input, and the teal frames are generated. The model keeps the identity and style of the input while adding motion such as turning, walking or camera drift. Identity drift over time is a good thing to check after quantization.

Almost all of the model's weight sits in three boxes: the video stream at 14 billion parameters, the audio stream at 5 billion, and Gemma-3 at 12 billion, about 31 billion in total. Read the diagram left to right: the prompt goes through Gemma, a feature extractor and two connectors, then both noisy latents are denoised together for N steps, and finally decoders turn latents into sound and frames. For text-to-audio-and-video only the decoders are needed, not the VAE encoders.

Peak memory depends on the tile size, not on the final 1080p output. The flow goes left to right: the full DiT first generates a base latent of about half a megapixel, a latent upscaler enlarges it, the result is cut into overlapping tiles in space and time, each tile is refined by the same DiT weights, and the tiles are blended before decoding. On a memory-limited Arc GPU this is what makes high resolution feasible.

The architecture is two parallel denoisers joined in the middle. Read it from the outside in: audio as a mel spectrogram and video each pass through their own VAE encoder and receive noise, while text passes through the encoder, feature extractor and one connector per stream. N dual-stream blocks then denoise both latents, exchanging information through audio-visual cross-attention, and each stream has its own prediction and loss. This is the training view of the component map above.

Cross-attention maps show that the model finds the source of each sound in the frame. The schematic shows four clips at three frames each, with V2A maps on one side and A2V maps on the other, averaged over heads and layers. Attention follows the passing car, switches from the first speaker to the second to both, and locks onto the lips in the close-up. This is the paper's qualitative evidence that the bidirectional design actually synchronises the two modalities.

Text conditioning uses every Gemma layer, not just the last one. Gemma-3 12B produces a tokens-by-width activation at each of its L layers, the feature extractor merges them into one embedding, and learned thinking tokens replace the padding positions. A small bidirectional transformer connector then refines the sequence for one stream, and there is one connector per stream. For quantization this means an error in early Gemma layers still reaches the DiT.

Modality guidance pushes the prediction along two independent directions. Start at the fully conditioned output M of x, t, m; move away from the text-dropped prediction by the weight s t, then away from the modality-dropped prediction by the weight s m. The end point is the guided output M prime. Arrow lengths are illustrative. It is the geometric picture of the guidance equation in section four point one.

One stream of one block, in full detail; audio and video share this layout. The latent passes through RMS norm, a timestep-driven scale and shift, self-attention with rotary positions, a gate and a residual add, then text cross-attention, then audio-visual cross-attention with the other branch, then a modulated and gated feed-forward network. The timestep AdaLN feeds every scale, shift and gate, and those small layers are the usual candidates to keep in high precision.

Rounding errors should not be thrown away but paid back. On the left, round to nearest, or RTN, rounds each weight alone, so the errors pile up and 3-bit OPT one hundred seventy-five B reaches a perplexity of about seven thousand. On the right, GPTQ rounds column j and then nudges the full-precision columns to its right, using the inverse Hessian built from calibration inputs. This one picture is the whole paper; everything later makes it accurate and fast.

Speed is not the only obstacle: the numbers themselves go bad at scale. Starting from a healthy, positive definite inverse Hessian, repeatedly applying the removal update of equation five lets round-off error build up until the matrix becomes indefinite. From then on the compensation pushes weights the wrong way and the layer is ruined. The paper says this almost certainly happens in a few layers of models beyond a few billion parameters, which motivates the Cholesky trick.

The first trick is to quantize columns in the same fixed order for every row. OBQ picks a greedy order per row, so each row keeps its own inverse Hessian to update, once per weight. Because the Hessian depends only on the inputs X, a shared order means all rows share one inverse, updated once per column. The paper finds arbitrary order is about as good as greedy on large layers, and the cost drops to the formula in the equations section.

Here is the core loop in motion. Each panel is one step t: column t, in indigo, is rounded, its error is scaled and multiplied by row t of the inverse Hessian, and that correction is subtracted from the teal columns to its right, which absorb it. Rust columns are already frozen. After the last column the whole matrix is quantized, giving Q. This is the picture behind Algorithm 1.

The second trick keeps the same arithmetic but reorders it for the GPU. Updating the whole weight matrix after every column is memory-bound, because each update touches everything but does little math. Since rounding column i only needs the updates to column i, GPTQ keeps updates inside a block of one hundred twenty-eight columns, then applies one large matrix multiply to the rest. Same operation count, about ten times faster in practice.

The third trick fixes the numerical drift from figure four. Quantizing column q only needs one row of the shrinking inverse Hessian, from the diagonal onward, and those rows are exactly the rows of the upper Cholesky factor, up to a square-root scale. So one stable Cholesky call, plus one percent dampening, gives every row in advance and replaces the error-prone chain of equation five. The error is then divided by the diagonal entry of U.

GPTQ never holds the whole model on the GPU. It loads one Transformer block of six linear layers, runs the calibration set through it to build a Hessian per layer, quantizes each layer, then re-runs the inputs through the now quantized block to feed the next one. Using inputs that already carry earlier quantization error gives a noticeable accuracy gain at almost no cost. For a diffusion transformer, the same loop would run over its blocks.

GPTQ tracks full precision across model sizes, while plain rounding falls apart. Each panel plots WikiText2 perplexity against parameter count on a log scale: OPT at four bits on the left and BLOOM at three bits on the right. The RTN curve spikes, to one hundred ten for OPT sixty-six B and five hundred seventy-one for BLOOM one hundred seventy-six B, while GPTQ hugs the FP16 line. It summarises tables three and four.

The paper's own sketch of the procedure matches Algorithm 1. On the right is the inverse Hessian in Cholesky form, computed once. On the left, a bold block of consecutive columns is quantized one column at a time, with the white column being the current one. The blue weights to the right are updated only once the block is finished. Compare it with the lazy-batch and Cholesky figures above.

Zero-shot task accuracy tells the same story as perplexity. The plots show LAMBADA accuracy against model size for OPT and BLOOM, where higher is better. At four bits even RTN mostly holds, except OPT sixty-six B. At three bits RTN drops to about zero, while GPTQ keeps most of the accuracy, and bigger models lose less. The data come from tables thirteen and fourteen.

Grouping closes most of the remaining four-bit gap. Curves show WikiText2 perplexity for OPT models from 1.3 to 30 billion parameters: plain row-wise four-bit GPTQ, groups of one thousand twenty-four, groups of one hundred twenty-eight, and FP16. The g128 curve sits closest to FP16. The grouped curves are redrawn from the plot, so read their shape, not exact values. This is the basis for int4 with group size one hundred twenty-eight in NNCF.