Written for a software engineer who knows tensors and basic linear algebra, but not model architecture. Every concept is built up before it is used.
Foundations — read these first if the terms are new
Strip away the mystique and a neural network layer is this:
y = x @ W.T + bx is your input vector, W is a matrix of learned numbers, b is a bias vector. That's it. That's a "linear layer" (also called "dense" or "fully connected", hence names like fc1 — fully connected 1).
Weights / parameters are the numbers in W and b. When people say "a 20 billion parameter model," they mean the total count of numbers across all its matrices. Training is the process of finding good values for them; inference (what you're doing) just uses them.
This is where your linear algebra pays off directly. In PyTorch a linear layer's weight has shape [out_features, in_features]. So:
fc1.weight = [28672, 5376]means: takes a 5376-element vector, produces a 28672-element vector. Every shape in this document can be read that way. When you see a chain of layers, you're watching a vector get reshaped step by step:
5376 ──fc1──> 28672 ──(gating)──> 14336 ──fc2──> 5376If you stack two matrix multiplies you get... another matrix multiply. (xA)B = x(AB). Stacking would be pointless. So between layers you apply a nonlinear function elementwise — something bent, like max(0,x) or a smooth S-curve. That bend is what lets stacked layers express things a single matrix can't.
Names you'll see: ReLU, GELU, SiLU/swish. They're all "a bend."
Every neural net has an input, an output, and intermediate values in between. The intermediates are called hidden states — "hidden" only in the sense of not being the input or output. Nothing mysterious.
hidden_size is the width of that intermediate vector. In your model it's 5376, meaning: at every point between blocks, each token is represented by 5376 floating-point numbers.
If it helps, think of it as a record size. The model's working data is:
[sequence_length, 5376] # one 5376-float row per tokenand every layer is a transformation on that array. That single shape is the spine of the whole architecture.
An important distinction for the memory discussion later:
What it is | Scales with | |
|---|---|---|
Weights | The learned matrices. Fixed after training. | Model size (constant at run time) |
Activations | The intermediate tensors flowing through during a forward pass. | How much data you're processing |
Your 20 GB of weights is constant no matter what you generate. Activations grow with the size of your video. That's why you OOM by asking for longer videos, not by loading a bigger model.
Running data through the network once, input → output. No learning, no gradients. All you ever do at inference time.
A token is one item in a sequence, represented as a vector.
Your data is a sequence of tokens, i.e. that [sequence_length, 5376] array. sequence_length (often written S) is how many tokens — and it is the single number that determines your compute cost.
A transformer repeats exactly two operations, over and over:
That's genuinely it. Attention mixes across the sequence; the MLP does per-token computation. A block (or layer) is one attention + one MLP. Your model stacks 50 of them.
tokens ─► [attn ─► MLP] ─► [attn ─► MLP] ─► ... ×50 ─► output
block 1 block 2The per-token part. Two matrices: widen, bend, narrow.
5376 ──fc1──> 14336 ──nonlinearity──> ──fc2──> 5376Why widen then narrow? The wide middle gives the nonlinearity more room to work — more dimensions in which to carve out useful distinctions before compressing back down. The widening factor is usually 3–4×.
SwiGLU — the variant your model uses — is a refinement. Instead of one wide output, fc1 produces two halves. One is passed through the bend and used as a gate, multiplied elementwise into the other:
gate, value = fc1(x).chunk(2)
hidden = swish(gate) * value # elementwise multiply
out = fc2(hidden)The gate lets the network dynamically suppress or amplify each dimension per token. This is why fc1 outputs 28672 = 2 × 14336 — double width, because it's producing two things. Costs 1.5× the parameters of a plain MLP and reliably works better.
Two supporting pieces you'll see everywhere:
x = x + block(x). Each block writes a correction onto a running total rather than replacing it. Without this, deep networks don't train. Think of it as the sequence accumulating edits.eps (epsilon) is a tiny constant in the denominator preventing division by zero.This is the one genuinely non-obvious idea, so here it is slowly.
Each token needs information from other tokens. A token representing a patch of frame 40 needs to know what was in frame 39, and what's happening elsewhere in frame 40. But which other tokens matter depends entirely on content — you can't hard-code it.
You know a hash lookup: exact key match, return one value.
value = table[key] # exact match, one resultAttention is that, made soft and differentiable. Instead of matching one key exactly, you compare your query against every key, get a similarity score for each, and return a weighted blend of all the values:
scores = query @ all_keys.T # how well do I match each token?
weights = softmax(scores) # normalize to sum to 1
result = weights @ all_values # weighted average of everythingSoftmax turns arbitrary scores into positive numbers summing to 1 — a weighting. High-scoring tokens dominate; low-scoring ones contribute nearly nothing.
Each token produces three vectors, each by its own learned matrix:
Name | Intuition | |
|---|---|---|
Q | Query | "What am I looking for?" |
K | Key | "What do I have to offer?" |
V | Value | "If you attend to me, here's what you get." |
Q and K are compared to compute relevance; V is what actually gets passed along. Keeping them separate means "how do I decide relevance" and "what do I contribute" are learned independently.
In your model these are computed by one packed matrix — qkv_proj of shape [21504, 5376], where 21504 = 3 × 7168. Q, K and V are produced in a single matrix multiply and then split. One large GEMM is faster than three small ones.
Doing this once forces the model to blend all kinds of relationships into one score. Instead it does it many times in parallel, each with different learned projections. Each parallel copy is a head.
Your model: 56 heads, each 128 wide. 56 × 128 = 7168 — that's where the 7168 comes from. Each head learns its own notion of relevance (one might track motion continuity, another spatial neighbours). Their outputs are concatenated and passed through out_proj [5376, 7168] to get back to the model width.
heads = how many parallel attentions.head_dim = how wide each one is.Every token compares against every other token. That's S × S comparisons — the cost grows with the square of sequence length. The MLP, by contrast, processes each token independently: cost grows linearly.
MLP: cost ∝ S
Attention: cost ∝ S²This one fact drives nearly every performance decision in this document.
Attention as described has no notion of order — it's a set operation. Position must be injected explicitly.
RoPE (Rotary Position Embedding) does this by rotating the Q and K vectors by an angle determined by position. Two tokens close together get similar rotations, so their dot product stays high; distant ones get rotated apart. Position becomes a geometric property of the vectors.
Your model uses 3D RoPE: each token has a position (t, h, w) — which frame, and where in that frame. It rotates by all three axes.
You could imagine a network that takes a prompt and emits all the pixels in one shot. In practice this produces mush. Generating a coherent video is too big a leap to make in one step.
Instead: start from pure random noise and refine it repeatedly, each pass making it slightly more like a real video. After 20–50 passes you have output. Each pass is a small, learnable step rather than one impossible jump.
That's diffusion. The network's job at each pass is: "given this partially-noisy thing, what should change?"
Something has to be the network doing the refining. Historically this was a U-Net (a convolutional architecture). Modern models use a transformer instead.
DiT = Diffusion Transformer. A transformer used as the denoiser in a diffusion model. That's the entire meaning of the acronym. Your model is a DiT.
Here's the framing that makes modern diffusion click.
Imagine every possible video is a point in a huge space. Pure noise is one region; real videos are another. Now imagine a field of arrows filling that space — at every point, an arrow saying "to become more like a real video, move this way." That is a vector field.
The model IS that field. You hand it a point (your current noisy latent) and a timestep, and it returns an arrow (a direction, called the velocity).
To generate, you follow the arrows in small steps:
x = random_noise
for t in schedule: # your "steps" setting
v = model(x, t, prompt) # which way should I move?
x = x + v * dt # take a small stepIf that looks like a game physics loop — pos += velocity * dt — that's exactly right. It's the same operation.
ODE stands for Ordinary Differential Equation: a rule that gives you a rate of change at each point, which you integrate to get a path. That's precisely what we have. Numerical integration just means "take lots of small steps instead of solving it exactly," and the simplest method — x += v * dt — is Euler's method.
So:
A sampler step is one step of numerical integration. More steps = smaller dt = less error, with diminishing returns.
This formulation is called flow matching (your log prints model_type FLOW). Older models used a noisier, probabilistic formulation (DDPM); flow matching is the cleaner modern version and needs far fewer steps.
t runs from 1 (pure noise) to 0 (finished). The schedule decides which values of t you actually visit.
Shift warps that schedule. Your model uses sigma_shift_video = 12.0, which spends disproportionately more steps near t = 1 (high noise). Why: early steps decide global structure — layout, motion, composition. Late steps polish texture. A structural mistake is unrecoverable; slightly soft texture isn't. So you buy resolution where it matters.
Refining raw pixels 20–50 times would be brutally expensive. Instead everything happens in a compressed representation called a latent, produced by a separate model (a VAE — covered in §8). Roughly 128× fewer numbers than pixels.
The diffusion model never sees a pixel. It refines latents; the VAE decodes at the end.
A trick to make output follow your prompt more closely. Run the model twice — once with your prompt, once with nothing — then extrapolate away from the promptless answer:
result = uncond + scale * (cond - uncond)This amplifies whatever your prompt actually changed. It costs 2× per step, because you're doing two forward passes instead of one.
From comfy/ldm/minimax/model.py:413:
hidden_size=5376, num_layers=50, num_attention_heads=56,
attention_head_dim=128, ffn_hidden_size=14336, ...You can now read every one of those. Working out one block's parameter count:
Matrix | Shape | What it does | Params |
|---|---|---|---|
| 21504 × 5376 | Makes Q, K, V (3 × 7168) | 115.6 M |
| 5376 × 7168 | Attention result → model width | 38.5 M |
| 28672 × 5376 | MLP widen (2× for SwiGLU gating) | 154.1 M |
| 5376 × 14336 | MLP narrow | 77.1 M |
per block | 385.4 M |
× 50 blocks ≈ 19.3 billion parameters, plus embeddings ≈ 20 B.
At int8 (one byte per parameter) that's ~19.3 GB — and your file on disk is 19.53 GB. The arithmetic closes, which is a good check that you've understood the structure.
For scale: this is comparable to a mid-large LLM. It barely fits on a 24 GB card, which is why all the memory machinery in §8 exists.
Three things happen, in order.
The model needs to know two things beyond the tokens themselves: what timestep we're at, and what you asked for.
The timestep enters through adaLN (adaptive Layer Normalization). Remember normalization rescales a vector. adaLN makes that rescaling conditional: a small network turns the timestep into scale, shift, and gate values applied around each norm.
So conditioning steers the model by modulating normalization — cheaper than adding extra attention layers, and empirically better for this job.
The mod_segments argument means this is applied per segment: video tokens and audio tokens get different modulation in the same pass.
56 heads × 128 dim, bidirectional, with 3D RoPE. As described in §3.
return comfy.ops.linear_input_act(self.fc2, self.fc1(x), "swiglu")fc1 widens 5376 → 28672, splits into gate and value, multiplies, fc2 brings 14336 → 5376.
LLM (e.g. LM Studio) | Video DiT | |
|---|---|---|
Attention | Causal — token n sees only < n | Bidirectional — all see all |
KV cache | Yes; each new token is cheap | None |
Per forward pass | 1 new token | The entire sequence |
Passes per output | 1 per token | steps × (2 if CFG) over everything |
KV cache is the LLM optimization where past K and V vectors are stored and reused, so generating token 500 doesn't redo tokens 1–499. It cannot apply here, because every token's value changes at every step. You recompute all 50 blocks over all tokens, 20–50 times over. This is why video generation is so much slower than chatting with a similarly sized LLM.
prompt + first/last reference frames
│
├─ tokenizer: "<Picture 1>: " <vision> "<Picture 2>: " <vision> <prompt>
│
▼
Qwen3-VL-32B (a 32B LLM used as a feature extractor) ──► [L, 5120]
│ (then EVICTED)
▼
condition_proj 5120→5376 + 2 refiner blocks
│
▼
┌──────────────────────────────────────────────────────────┐
│ DENOISING LOOP × steps × (2 if CFG) │
│ │
│ ONE sequence, tagged by segment: │
│ [ text | cond | ref_img | video | ref_audio | audio ] │
│ │
│ 50 × block: adaLN → attention → SwiGLU MLP │
│ → velocity → x += v·dt │
└──────────────────────────────────────────────────────────┘
│
├─► Video VAE decode → frames
└─► Audio VAE decode → waveformThe structurally important part — model.py:588-599:
h = torch.empty(layout.seq_len, self.hidden_size, ...)
for a, b, kind in layout.segments:
if kind == "text": h[a:b] = text_states
elif kind in ("cond", "ref_img", "video"): h[a:b] = video_embed[...]
else: # ref_audio / audio h[a:b] = audio_embed[...]Text, reference images, video latents and audio latents all live in one tensor. Attention runs across the whole thing — which is how the model syncs a mouth movement to a phoneme: those tokens literally attend to each other.
It also means anything you add to the conditioning makes every step slower, because it permanently increases S.
.safetensors is deliberately boring: an 8-byte header length, a JSON header, then raw tensor bytes. No pickle, so no code execution — which is why the ecosystem abandoned .ckpt. Metadata is free to read:
n = struct.unpack("<Q", f.read(8))[0]
hdr = json.loads(f.read(n)) # shapes, dtypes, byte offsetsLoading is memory-mapped (mmap): the OS maps the file into the address space and pages in bytes on demand, zero-copy.
There's no manifest. model_detection.py infers the architecture from tensor names and shapes, matching against candidates in supported_models.py. Yours matches {"image_model": "minimax_h3"}, which selects shift: 12.0 and memory_usage_factor: 0.114 (a heuristic for predicting activation memory — which is exactly why an OOM can surprise it).
Your prompt is turned into vectors by Qwen3-VL-32B — a 32-billion-parameter vision-language model. Three things about how it's used are unusual:
It's truncated and headless. The checkpoint stops at layer 50, and the conditioning is the raw hidden state there with no final normalization. The part of an LLM that predicts words is discarded entirely. You're using it purely as a semantic feature extractor — its internal representation of meaning, not its output.
No chat template. From the module docstring: The H3 presentation is NOT chat-templated. Raw token ids, no system prompt, none of the <|im_start|> scaffolding.
It handles your reference images. fl2va = first/last frame → video + audio. The tokenizer builds one interleaved sequence:
"<Picture 1>: " <vision block> "<Picture 2>: " <vision block> <your prompt>Your frames go through Qwen's vision tower (the image half of the VLM) into the same token stream as your words, so images and text are cross-referenced before the diffusion model sees anything.
Why 14.6 GB is affordable: it runs once, then is evicted. Compare with the DiT running 50 blocks over ~37,000 tokens, 40+ times.
A Variational Autoencoder is two networks trained together:
The variational part matters. A plain autoencoder learns an arbitrary, pothole-ridden latent space — interpolate between two points and you get garbage. The VAE adds a KL penalty (KL divergence measures distance between probability distributions) pulling the latent distribution toward a standard Gaussian. The result is smooth and continuous: nearby points decode to similar images.
That smoothness is the precondition for diffusion. The sampler traces a continuous path through latent space, so the space has to be well-behaved everywhere along it.
The VAE is trained separately and frozen.
Your compression (latent_formats.py:570):
spacial_downscale_ratio = 16 # a 16×16 pixel block → 1 latent position
temporal_downscale_ratio = 4 # nominal (real value is 3.4 — see §9)
latent_channels = 24For one second of 1280×704 @ 24 fps:
positions | values | |
|---|---|---|
Pixels | 24 × 1280 × 704 | 64.9 M (×3 channels) |
Latent | 6 × 44 × 80 | 507 K (×24 channels) |
~128× fewer numbers.
Two VAEs, because the modalities are unrelated: video uses 3D convolutions over space and time; audio uses a 1D waveform VAE. Your audio VAE loads in float32 while video is fp16 — phase errors in audio are far more audible than a slightly-off pixel.
The latent is a 3D grid. To feed a transformer it must become a sequence. Patchify groups neighbouring latent cells into one token: patch_size = (1,2,2) means no temporal grouping, 2×2 spatial. So four latent positions × 24 channels = 96 numbers become one token, projected up to 5376.
latent_rgb_factorsA linear approximation of the VAE decoder — 24 latent channels → 3 RGB, one matrix multiply. Gives a rough live preview without running the real decoder. Enable with --preview-method latent2rgb; it's nearly free.
1. Stage text encoder (15 GB) → encode → EVICT
2. Stage DiT (20 GB) → denoising loop
3. Stage VAEs (5.5 GB) → decode15 + 20 + 5.5 = 40 GB against a 24 GB card, so these must be sequential. Underneath, DynamicVRAM (comfy-aimdo) streams weights within each stage from pinned system RAM, prefetching block n+1 while block n computes. Your 128 GB of RAM is what makes this viable at all.
Spatially it's a clean divide-by-32 (16 for the VAE, 2 for the patch). Temporally it is not — the node snaps frames onto a 17-frames-to-5-latents grid (nodes_minimax_h3.py:33-40):
def align_frame_count(n):
while n % 17 != 5:
n += 1
def video_latent_t(frame_count):
return 2 if frame_count <= 5 else ((frame_count - 5) // 17) * 5 + 2Real temporal compression is 17:5 = 3.4×, not the nominal 4 in latent_formats.py.
╔═══════════════════════════════════════════════════════╗
║ latent_t = ((frames - 5) // 17) × 5 + 2 ║
║ tokens = latent_t × (height / 32) × (width / 32) ║
╚═══════════════════════════════════════════════════════╝Resolution | Frames | latent_t | Tokens |
|---|---|---|---|
832 × 480 | 90 | 27 | ~10,500 |
1280 × 704 | 124 | 37 | ~32,600 |
1344 × 768 (node default) | 124 | 37 | ~37,300 |
1280 × 704 | 243 | 72 | ~63,400 |
1920 × 1088 | 124 | 37 | ~75,500 |
Component | Scales as | Why |
|---|---|---|
MLP + projections | O(S) | Fixed work per token |
Attention | O(S²) | Every token attends to every token |
Activation memory | O(S) | Flash/Sage keep attention memory linear |
Weight memory | O(1) | Always 20 GB |
Measured on your 4090, after our optimizations (per forward pass, all 50 blocks):
Tokens | MLP + proj | Attention | Total |
|---|---|---|---|
16,384 | ~1.9 s | ~1.5 s | ~3.4 s |
32,768 | ~3.7 s | ~5.8 s | ~9.5 s |
Doubling tokens doubled the linear cost and quadrupled attention. The crossover where attention overtakes everything else sits around 22–25k tokens.
Consequence: doubling both resolution dimensions = 4× tokens = 4× linear but 16× attention. Doubling length = 2× tokens = 2× and 4×. Length is much cheaper than resolution for the same added value.
Resolution — your most expensive knob. Consider generating smaller and upscaling afterward (you have SeedVR2 installed for exactly this).
Length / frames — linear in tokens, so still quadratic in attention, but a gentler climb than resolution.
Steps — a pure linear multiplier. ODE integration steps; flow-matching models hold up well at lower counts. The cheapest dial to lower.
CFG — a flat 2× on the entire loop. If your workflow tolerates CFG 1.0, that's an instant halving.
Shift — free. Reshapes where steps are spent, not how many. Tune this before spending more steps.
Reference images — cost you twice: as Qwen vision tokens (W/32) × (H/32), and as VAE-encoded condition rows. Both ride every step — the node docstring says condition latents are "re-injected every step (never denoised)".
You do not need a resize node upstream.MiniMaxH3ImageToVideo(nodes_minimax_h3.py:130-139) already resizes keyframes to the node's ownwidth/heightbefore Qwen or the VAE sees them. A 4K reference is downsampled on arrival.
>
But watch aspect ratio.first_frameusescrop="disabled"— a plain stretch that distorts if your reference's aspect doesn't matchwidth/height.last_frameusescrop="center"and is safe. Put an aspect-preserving crop beforefirst_frameif needed — for geometry, not speed.
At the default 1344 × 768, each keyframe adds ~1,008 Qwen tokens + ~1,008 latent rows; first + last ≈ 4,032 tokens on a ~37,300 base.
For ref2va, MiniMaxH3ReferenceToVideo exposes ref_image_size: "match" (default) or "max" (2048 px short edge). Its tooltip warns "max" "can be several times slower." That's your cost/fidelity dial there.
Node defaults — 1344 × 768, 124 frames, 20 steps, CFG on:
tokens ≈ 37,300 video + ~400 audio + ~4,000 keyframes ≈ 42,000
attention ≈ 189 ms/block × 50 ≈ 9.4 s (S² scaling from the 32,768 measurement)
linears ≈ 4.8 s
per pass ≈ 14 s
× 2 CFG ≈ 28 s / step
× 20 ≈ 560 s (~9 min) plus text encode + VAE decodeDrop to 832 × 480 and tokens fall to ~10,500 — attention drops roughly 16×.
Change | Effect | Measured |
|---|---|---|
torch cu126 → cu130 | Re-enabled fused int8 GPU kernels | 4.1–6.0× on MLP matrices |
Same | Rescaling moved inside the matmul | fc1 peak VRAM 7.98 → 2.40 GB |
SageAttention + Triton | int8-quantized attention | 2.6× on attention |
comfy/quant_ops.py:40 disables the compiled GPU kernels when torch.version.cuda < 13. The Python fallback materialized a full int32 result, looped over it converting chunks to float, then concatenated a second full copy — roughly three extra full-tensor round-trips to VRAM per matrix, across ~200 matrices per step. The fused kernel does the rescaling inside the matmul; the intermediate never touches memory.
convrot in the filename meansQuantization stores weights at lower precision (int8 = one byte instead of four) to save memory and time. It fails naively on transformers because activations have outlier channels — a few dimensions 100× larger than the rest. Since the scale must cover the maximum, outliers consume the whole numeric range and everything else collapses toward zero.
The fix: multiply activations by a Hadamard matrix (an orthogonal matrix of ±1) before quantizing. It mixes every channel into every other, smearing outlier energy across the group so the distribution becomes roughly Gaussian — which int8's uniform grid handles well. Being orthogonal it's exactly invertible, and the inverse folds into the weights for free. The linear algebra is unchanged; only the numerical conditioning improves.
Term | Meaning |
|---|---|
Parameter / weight | A learned number. "20 B parameters" = 20 billion of them. |
Linear / dense / fully-connected layer | A matrix multiply plus bias: |
| "Fully connected" — the MLP's two matrices. |
Shape | PyTorch weight convention. |
Activation function | The elementwise "bend" between layers (ReLU, GELU, SiLU/swish). Without it, stacked layers collapse into one. |
Hidden state | An intermediate vector — not input, not output. |
| Width of that vector. 5376 here. |
Activation (tensor) | Intermediate data flowing through a forward pass. Scales with input size. |
Forward pass | Running data through once, input → output. |
Residual / skip connection |
|
Normalization / RMSNorm | Rescaling a vector to consistent magnitude. RMSNorm divides by root-mean-square. |
| Tiny constant preventing division by zero inside a norm. |
MLP / FFN | The per-token widen-bend-narrow stage. |
GLU / SwiGLU | Gated variant: split the output, use one half to gate the other. Why |
Term | Meaning |
|---|---|
Token | One item in the sequence, as a vector. Here: a patch of compressed video. |
Sequence length (S) | How many tokens. Governs your cost. |
Block / layer | One attention + one MLP. Your model has 50. |
Attention | Tokens exchanging information — a soft, weighted dictionary lookup. |
Q / K / V | Query ("what I want"), Key ("what I offer"), Value ("what you get"). |
| One matrix producing all three at once. |
| Projects the attention result back to model width. |
Softmax | Turns scores into positive weights summing to 1. |
Head | One parallel attention computation. Yours: 56. |
| Width of each head. 128. 56 × 128 = 7168. |
Self-attention | Tokens attend within their own sequence. |
Causal | A token sees only earlier ones (LLM generation). |
Bidirectional | Every token sees every other. What your model uses. |
KV cache | LLM speedup storing past K/V. Not applicable here. |
RoPE | Rotary Position Embedding — encodes position by rotating Q and K. Yours is 3D over (t, h, w). |
Patchify | Grouping neighbouring latent cells into one token. |
Transformer | Architecture of repeated attention + MLP blocks. |
DiT | Diffusion Transformer — a transformer used as a diffusion model's denoiser. |
Term | Meaning |
|---|---|
Diffusion | Generating by starting from noise and refining repeatedly. |
Denoiser | The network doing the refining. |
Latent / latent space | Compressed representation the model works in instead of pixels. |
VAE | Variational Autoencoder — encoder/decoder between pixels and latents. |
KL divergence | A distance between probability distributions. The VAE's KL penalty keeps latent space smooth. |
Vector field | An arrow at every point in space. The model is one. |
Velocity | The arrow the model returns — which way to move. |
ODE | Ordinary Differential Equation — a rule giving rate of change; integrate it to get a path. |
Numerical integration | Approximating that path with small steps. |
Euler's method | The simplest one: |
Flow matching | Modern formulation: learn the velocity field, integrate the ODE. Your log: |
DDPM | The older, probabilistic formulation. Needed more steps. |
Timestep / sigma | Position on the noise→data path. 1 = noise, 0 = done. |
Steps | Integration steps. More = finer, diminishing returns. |
Scheduler | Chooses which timesteps to visit. |
Sampler | The integration algorithm (Euler, DPM++, …). |
Shift | Warps the schedule toward high noise. 12.0 for video here. |
CFG | Classifier-Free Guidance — run with and without the prompt, extrapolate. Costs 2×. |
Conditioning | Everything steering generation: prompt, reference frames, audio. |
adaLN | Adaptive LayerNorm — inject conditioning by modulating normalization. |
Text encoder | Turns your prompt into vectors. Here: Qwen3-VL-32B. |
LLM / VLM | Large Language Model / Vision-Language Model. |
Vision tower | The image-processing half of a VLM. |
Tokenizer | Splits text into integer ids. |
CLIP | An older text/image encoder. ComfyUI still labels the text-encoder slot "CLIP" for historical reasons. |
Term | Meaning |
|---|---|
Quantization | Storing/computing at lower precision to save memory and time. |
fp32 / fp16 / bf16 | 32- and 16-bit floats. bf16 trades precision for fp32's range. |
fp8, int8, nvfp4 | 8-bit float, 8-bit integer, 4-bit float formats. Need scale factors. |
AWQ | Activation-aware Weight Quantization — protects the weights that matter most. |
Outlier channels | The few huge-valued dimensions that wreck naive quantization. |
Hadamard matrix / | Orthogonal ±1 matrix used to spread outliers so int8 works. Invertible, folds into weights. |
Dequantize | Converting back to float using the scale. |
Term | Meaning |
|---|---|
VRAM | GPU memory. Yours: 24 GB. |
Kernel | A function that runs on the GPU. |
GEMM | GEneral Matrix Multiply — the core operation. |
Epilogue | Work fused onto the end of a GEMM (scaling, bias) so intermediates never hit memory. Where our 4–6× came from. |
Fused kernel | One kernel doing several steps, avoiding memory round-trips. |
Memory-bandwidth-bound | Limited by moving data, not by arithmetic. Most inference is. |
FLOPs / TFLOPS | Floating-point operations; trillions per second. |
CUDA | NVIDIA's GPU programming platform. |
cuDNN / cuBLAS / CUTLASS | NVIDIA libraries for neural-net ops, linear algebra, templated GEMMs. |
Triton | Language for writing GPU kernels in Python, compiled on first use. |
JIT | Just-In-Time compilation. Cached in |
SDPA | Scaled Dot-Product Attention — PyTorch's built-in attention. |
Flash Attention | Attention that never materializes the S×S matrix, making memory O(S) not O(S²). |
SageAttention | Quantized attention (int8 Q/K). 2.6× here, slightly lossy. |
safetensors | Weight file format: JSON header + raw bytes, no code execution. |
State dict | The |
mmap | Memory-mapped file — OS pages it in on demand, zero-copy. |
Pinned memory | Page-locked RAM that can't be swapped, enabling fast DMA to GPU. Yours: 52 GB. |
PCIe | The bus between RAM and GPU. The bottleneck when streaming weights. |
Offloading | Keeping weights in system RAM, moving to VRAM only when needed. |
DynamicVRAM / comfy-aimdo | ComfyUI 0.30's automatic block-streaming system. |
Prefetch | Fetching block n+1's weights while block n computes. |
OOM | Out Of Memory. |
Tiled decode | Decoding the VAE in overlapping chunks to cap peak VRAM. |
Name | Meaning |
|---|---|
| The running hidden state — the |
| Timestep embedding — current noise level as a vector. |
| Which span is video / audio / text, so adaLN can treat them differently. |
| Precomputed rotation table for positions. |
|
|
| Total token count (S). |
| Latent channels — 24 video, 32 audio. |
|
|
| Per-modality schedule warp (12.0 / 3.0). |
| VAE spatial compression (16). (sic — misspelled in ComfyUI.) |
| Heuristic for predicting activation memory. |
| Linear latent→RGB approximation for previews. |
| Boolean mask: which rows are generated vs. held fixed as conditioning. |
| first-last / reference / text → video+audio. Yours is |
| Layers removed from the original release to shrink it. |
| Compiled C++/CUDA extension. |
Written during a debugging session, 2026-08-06. Measurements from your RTX 4090; architecture read from your installed ComfyUI 0.30.0 source and model files.