CARE Leaderboard

Leaderboard

Sort and filter all strategy × model × config results. Rankings change with metric choice — that's the point.

CARE Cross-Model Summary (Tables 11 & 20)

All models evaluated at 256 nominal steps, English-filtered, OWT prefix=64, gen=128, t=0.25. MAUVE gap = none minus high_entropy (positive = none wins). HumanEval pass@1 at t=0.0, compute-matched.

Pareto Explorer

PPL vs MAUVE trade-off. Filled = deterministic (t=0.0); hollow = stochastic. Point size ∝ NFE.

Show Pareto Front: Highlight Stochastic:

LLaDA-8B-Base — MAUVE vs PPL

Stochastic configs dominate the frontier — higher MAUVE at every PPL level.

Dream-7B-Base — MAUVE vs PPL

Best strategy changes at every budget level — confirming metric and budget dependence on an independent model family.

The Pareto Insight

On LLaDA, stochastic configurations dominate the frontier — achieving higher MAUVE at every PPL level. On Dream, the "best" strategy depends on both the step budget and which metric you prioritize. There is no single winner.

Stochasticity Lab

t = 0.0 (deterministic)

t = 0.1 (mild stochasticity)

t = 0.25 (moderate)

t = 0.5 (high) — ⚠ Collapse zone

Explore: Set unmask_temp:

0.00.10.250.5

⚠ Collapse zone: high_entropy remasking significantly reduces MAUVE quality at this temperature.

Strategy	MAUVE at selected t	NFE (128 steps)

MAUVE Gap Heatmap — none minus high_entropy (positive = none wins)

Starred cell: 256 steps, t=0.25 — gap=0.296±0.042, t-stat=7.06, p=0.020. Color scale: light teal (gap≈0) → red (gap≥0.4).

Self-BLEU vs Temperature (256 steps)

Lower = more pairwise diversity. high_entropy consistently most diverse — yet worst MAUVE.

Distinct-3 vs Temperature (256 steps)

high_entropy vocabulary collapses at t≥0.25 — explaining distributional collapse despite low Self-BLEU.

PPL vs Temperature — The Fluency Trade-off (Table 16)

Stochasticity improves MAUVE (see heatmap above) but degrades fluency. The two metrics tell opposite stories — single-metric evaluation hides this trade-off.

none strategy — 3 step budgets

high_entropy strategy — 3 step budgets

PPL via Llama-3-8B, LLaDA-8B-Base, English-filtered. Lower = more fluent. none degrades steeply with stochasticity; high_entropy shows less PPL sensitivity but worse absolute fluency throughout.

Three-Way ANOVA — Effect Sizes (Table 8)

Temperature dominates (η²=0.908). The Strategy × Temperature interaction (η²=0.469, p=0.002) confirms that strategy choice cannot be evaluated without specifying the stochasticity regime.

Compute Audit

What "256 steps" actually costs in forward passes (NFE) — 1× to 4× variation hidden in plain sight.

Actual NFE at 256 Nominal Steps — by Strategy

Any paper reporting improvement at 256 steps without actual NFE is making an unfair comparison. Dashed line = 256 nominal steps.

NFE Budget Equalizer

At equal compute, how many steps can each strategy run?

Target NFE:

How Conclusions Change Under CARE (cf. Table 4)

Without Controlled Evaluation	Under CARE
Strategy X improves at 256 steps	X uses 3.4× more compute; improvement vanishes at matched NFE
Strategy Y achieves best quality	Y wins PPL but loses MAUVE — metric-dependent
Informed remasking improves generation	At high steps + stochastic unmasking, it REDUCES quality by 0.30 (p=0.020)
Our remasking gains MAUVE by 0.15	t: 0.0→0.1 gains 0.3 MAUVE — for free, 2–4× less compute

Causal Evidence: Remask Fraction Ablation (Table 9)

MAUVE degrades monotonically as remask fraction increases — confirming that high_entropy's quality collapse is caused by persistent token churn, not evaluation noise. Single seed, high_entropy@256, t=0.25.

Compute-Matched Quality Comparison (Table 4)

At identical NFE budget, none outperforms all remasking strategies on MAUVE. PPL remains higher (worse) for all remasking strategies across all compute levels.

Model Deep-Dive

Per-model results, configs, and reproduction commands — all 12 evaluated models.

Tier 1 — Primary Evaluation

LLaDA-8B-Base

Dense8B paramsOpen weights Masked Diffusion LM · LLaMA-3-8B

Paper coming soon HF Model coming soon

MAUVE jumps from 0.663 (t=0.0) to 0.976 (t=0.1) with no-remasking — the single largest improvement observed across all strategies.

Best MAUVE-efficient config: none@64 steps, t=0.25 → MAUVE=0.984. This achieves near-peak quality at 1/4 the NFE of high_entropy@256.

Deterministic results (Table 5) — stochastic sweep in Stochasticity Lab tab.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3 (key comparisons), 1 (ablations)
💻 Hardware: NVIDIA A100 80GB, total ~500 GPU-hours
📊 Metrics: MAUVE, PPL (GPT-2 XL), Self-BLEU, Distinct-3, NFE

$ care-eval run \
    --model llada-8b-base \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

GSM8K accuracy (%) — LLaDA-8B-Instruct, 4-shot, n=200, SE≈3.5%. No significant differences across configurations confirm NFE confounds extend to code generation (Table 18).

HumanEval pass@1 (%) — Table 11

n=100, 3-seed mean±SE via lm-eval. Nominal-step comparison favors high_entropy (31.7 vs 28.3), but at matched NFE=437 the compute-matched none* achieves 32.0 — reversing the ranking. The NFE confound extends to code generation.

* none adjusted to higher nominal steps to match NFE=437. Highlighted = highest pass@1 per block.

LLaDA-8B-Instruct

Dense8B paramsOpen weightsInstruction-tuned · LLaMA-3-8B

Instruction tuning does not remove the stochasticity confound — MAUVE gap 0.287 (none vs. high_entropy) nearly matches the base model (0.296).

Highest HumanEval among LLaDA variants at 36.5 pass@1. MAUVE 0.928 under none@256, t=0.25. Confirms that instruction fine-tuning does not mitigate the remasking × stochasticity interaction identified in CARE.

CARE Tier 1 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3 (key comparisons), 1 (ablations)
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL (GPT-2 XL), Self-BLEU, Distinct-3, NFE

$ care-eval run \
    --model llada-8b-instruct \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

LLaDA-1.5

Dense8B paramsOpen weightsLLaMA-3-8B · builds on LLaDA-8B-Instruct

Best HumanEval in Tier 1 at 38.0 pass@1 — architecture improvements raise code performance while the stochasticity confound (gap=0.258) persists.

Updated LLaDA architecture built on LLaDA-8B-Instruct (foundation: LLaMA-3-8B). Improved MAUVE under none (0.939) and a modestly smaller gap vs. LLaDA-8B-Base (0.258 vs. 0.296).

CARE Tier 1 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3 (key comparisons), 1 (ablations)
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL (GPT-2 XL), Self-BLEU, Distinct-3, NFE

$ care-eval run \
    --model llada-1.5 \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

LLaDA-MoE-7B-A1B

MoE~1B active paramsOpen weightsTrained from scratch · ~20T tokens

Cross-architecture validation: stochasticity replicates — +0.507 MAUVE at t=0.1 vs t=0.0 (none@256).

Dense 8B vs MoE ~1B active — both models show the same stochasticity × remasking interaction pattern, confirming CARE findings are architecture-agnostic. Results at t=0.1 (optimal for MoE). Trained from scratch on ~20T tokens; no third-party LLM foundation.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🌡️ Temperature: t=0.1 (optimal for MoE architecture; Tier 1 results at t=0.1)
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 1 (MoE ablation study)
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE (primary), PPL (secondary)

$ care-eval run \
    --model llada-moe-7b-a1b \
    --strategy none high_entropy \
    --steps 256 \
    --unmask_temp 0.0 0.1 0.25 \
    --seeds 1 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

Dream-7B-Instruct

Independent family7B paramsOpen weightsInstruction-tuned · Qwen2.5-7B

Small MAUVE gap of 0.066 — Dream-family robustness to the stochasticity confound carries over to the instruction-tuned variant.

HumanEval 31.0 pass@1. MAUVE 0.927 under none@256, t=0.25. Dream's architecture is inherently more robust to remasking strategy confounds — instruction tuning preserves this advantage while boosting code performance.

CARE Tier 1 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3 (key comparisons), 1 (ablations)
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL (GPT-2 XL), Self-BLEU, Distinct-3, NFE
🎯 Strategies: entropy, origin, maskgit (Dream-family naming)

$ care-eval run \
    --model dream-7b-instruct \
    --strategy entropy origin maskgit \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

Dream-7B-Base

Independent family7B paramsOpen weightsQwen2.5-7B

Best strategy changes at every budget level — metric dependence is extreme on Dream. Smallest Tier 1 gap at 0.034.

maskgit t=0 achieves best PPL but worst MAUVE consistently. entropy@128 t=0.0 achieves MAUVE=0.977 — the highest single result across all Dream configs. Dream's architecture is inherently more robust to remasking strategy confounds than LLaDA variants.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3 (key comparisons), 1 (ablations)
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL (GPT-2 XL), Self-BLEU, Distinct-3, NFE
🎯 Strategies: entropy, origin, maskgit (Dream-family naming; entropy ≈ high_entropy)

$ care-eval run \
    --model dream-7b-base \
    --strategy entropy origin maskgit \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

Tier 2 — Extended Coverage

SM-Dream-7B

Tier 27B paramsOpen weightsSoft-Masked variant · Qwen2.5-7B via Dream-7B-Base

Soft masking variant of Dream — gap of 0.090 is larger than Dream-7B-Base (0.034) but smaller than LLaDA variants, suggesting soft masking partially reduces stochasticity sensitivity.

MAUVE 0.913 (none) vs 0.823 (high_entropy) at t=0.25, 256 steps. HumanEval 29.5 pass@1. Evaluated as an architecture variant study — same base as Dream-7B but with soft token masking during training.

CARE Tier 2 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL, HumanEval pass@1

$ care-eval run \
    --model sm-dream-7b \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

Dream-Coder-7B

Tier 27B paramsOpen weightsCode specialization · Qwen2.5-Coder-7B

Best HumanEval in Tier 2 at 34.0 pass@1 — code specialization pays off. Moderate gap (0.144) vs Dream-7B-Base (0.034) suggests code fine-tuning increases stochasticity sensitivity.

MAUVE 0.886 (none) vs 0.742 (high_entropy) at t=0.25, 256 steps. Demonstrates that code-specialized models retain the remasking confound despite domain adaptation.

CARE Tier 2 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL, HumanEval pass@1

$ care-eval run \
    --model dream-coder-7b \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix humaneval

care-eval CLI coming soon

DiffuLLaMA-7B

Tier 27B paramsOpen weightsLLaMA-2-7B

Gap of 0.255 despite 7B scale — older LLaMA-2 base contributes to stronger stochasticity sensitivity than newer Qwen2.5-based models at equivalent scale.

MAUVE 0.872 (none) vs 0.617 (high_entropy) at t=0.25, 256 steps. HumanEval 21.0 pass@1. One of the earliest diffusion-adapted LLMs; provides historical baseline for architecture evolution in dLLMs.

CARE Tier 2 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3
💻 Hardware: NVIDIA A100 80GB
📊 Metrics: MAUVE, PPL, HumanEval pass@1

$ care-eval run \
    --model diffullama-7b \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

Tiny-A2D (Qwen-0.6B)

Tier 20.6B paramsQwen3-0.6B

Sub-1B scale: MAUVE 0.691 at t=0.25 — well below 7B models, confirming that scale is a meaningful factor for dLLM generation quality.

Gap 0.215 remains significant at small scale. Marginal improvement over 0.5B sibling (MAUVE 0.691 vs 0.662). Part of the Tiny-A2D scale ablation study — shows the stochasticity confound persists even at minimal parameter counts.

CARE Tier 2 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3
💻 Hardware: NVIDIA A100 80GB (or A10/A6000 — small model)
📊 Metrics: MAUVE, PPL, HumanEval pass@1

$ care-eval run \
    --model tiny-a2d-qwen-06b \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

Tiny-A2D (Qwen-0.5B)

Tier 20.5B paramsQwen2-0.5B

Smallest dLLM evaluated: MAUVE 0.662 — scale floor for dLLM generation quality. Gap 0.214 nearly identical to 0.6B sibling, confirming the confound is scale-invariant below 1B params.

HumanEval 4.5 pass@1. Lowest MAUVE among all dLLM models in the CARE leaderboard. The near-identical gap between the 0.5B and 0.6B Tiny-A2D variants (0.214 vs 0.215) suggests the remasking × stochasticity interaction magnitude saturates below 1B params.

CARE Tier 2 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3
💻 Hardware: NVIDIA A100 80GB (or A10/A6000 — small model)
📊 Metrics: MAUVE, PPL, HumanEval pass@1

$ care-eval run \
    --model tiny-a2d-qwen-05b \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

BERT-Chat

Tier 2150M paramsModernBERT-base · MDLM SFT (dLLM framework)

Smallest model in CARE at 150M params: MAUVE 0.418 reflects scale limits. Remasking confound (gap=0.135) persists even at this scale and architecture type.

ModernBERT-base-chat-v0.1 — SFT-ed directly on ModernBERT-base via MDLM without generative pretraining (dLLM framework, Zhou et al. 2025). HumanEval 0.8 pass@1. Provides architecture diversity baseline: the CARE stochasticity confound appears in modernized encoder-based masked LMs, not only decoder-based dLLMs.

CARE Tier 2 results — 256 nominal steps, English-filtered OWT, t=0.25.

📋 Setup: 64-token OpenWebText prefix → generate 128 tokens
🔤 English filter: ASCII ≥ 0.85, CJK ≤ 0.02
🌱 Seeds: 3
💻 Hardware: NVIDIA A10 / A6000 (150M requires minimal VRAM)
📊 Metrics: MAUVE, PPL, HumanEval pass@1
🏗️ ModernBERT-base (149M) SFT-ed with MDLM on tulu-3-sft-mixture + smoltalk
⚠️ Encoder-based: no generative pretraining step — MDLM SFT applied directly to base checkpoint

$ care-eval run \
    --model bert-chat \
    --strategy none high_entropy \
    --steps 64 128 256 \
    --unmask_temp 0.0 0.1 0.25 0.5 \
    --seeds 3 \
    --tasks openwebtext_prefix

care-eval CLI coming soon

CARE Evaluation Protocol

Seven practices that prevent contradictory conclusions. Click any card to expand.

Framework Comparison (Table 2)

Framework	Multi-metric	NFE	Stoch. ctrl	Remask	Interaction
HELM	✓	—	—	—	—
dLLM	✓	partial	partial	—	—
Gen. Frontiers	PPL-ent.	implicit	implicit	—	—
ReMDM	✓	✓	partial	partial	—
CARE (ours)	✓	✓	✓	✓	✓

Three-Seed Reproducibility (Table 13)

MAUVE across 3 independent seeds. Highlighted rows (SE ≥ 0.030) show where single-seed conclusions are unreliable — directly motivating Protocol 6 (Report Uncertainty).

Language Filter Acceptance Rates — OWT (Table 19)

Acceptance rates (%) after applying the CARE language filter (ASCII ≥ 0.85, CJK ≤ 0.02). High stochasticity + aggressive remasking produces the most off-target language — dropping to 90% acceptance at t=0.5. LM1B stays 98%+ throughout, confirming the filter detects real multilingual leakage rather than noise (Table 20).

Submit Your Model

Run the official CARE harness and submit your results.

1
Install
```
$ pip install care-eval
```
2
Configure — create model_config.json (schema below)

Run

$ care-eval run --config model_config.json

Evaluate

$ care-eval metrics --config model_config.json --outputs ./outputs/

Bundle

$ care-eval bundle  →  generates submission.json

6
Submit — upload submission.json coming soon

🥉

Paper-reported

Metrics transcribed from paper. Not independently verified.

🥈

CARE-harness self-evaluated

Computed using official CARE harness on submitter's hardware. Raw outputs included.

🥇

Verified by maintainers

CARE harness re-run on reference A100. All outputs match submitted results.

Submission Schema

Paper Figures

All figures from the CARE paper with exact captions. Static companion view — click "View interactive" to explore the live data version in its dedicated tab.

1 Overview & Motivation

CARE end-to-end workflow: from model config to multi-metric evaluation and leaderboard submission

Figure 1CARE Evaluation Workflow — End-to-End Overview

The CARE protocol takes a model config, runs generation under controlled NFE, stochasticity, and remasking conditions, applies the seven-point evaluation checklist (NFE accounting, multi-metric reporting, stochasticity sweep, language filtering, none baseline, uncertainty quantification, external-evaluator PPL), and produces a reproducible submission JSON for the leaderboard.

Problem statement: same nominal steps hide 1×–4× NFE variation

Figure 1aThe Hidden Compute Problem

Remasking strategies share the same nominal step count but differ by up to 4× in actual forward passes. Without NFE control, quality improvements may simply reflect more compute.

NFE variation across strategies at 256 nominal steps

Figure 1bNFE Variation at 256 Nominal Steps

none=128, random=257, high_entropy=437, running_confidence=513 NFE at 256 nominal steps. The 4× spread is invisible without CARE's NFE accounting.

CARE framework comparison with HELM, dLLM, Gen. Frontiers, ReMDM

Figure 1cFramework Comparison

CARE is the only framework controlling NFE, stochasticity, remasking strategy, and their interaction jointly. Prior frameworks address at most two dimensions.

2 Pareto & Frontier Analysis

LLaDA-8B-Base PPL vs MAUVE Pareto frontier across strategies and temperatures

Figure 2LLaDA-8B-Base PPL–MAUVE Pareto Frontier

Filled markers = deterministic (t=0.0); open markers = stochastic (t=0.25). PPL via Llama-3-8B; MAUVE on OpenWebText, English-filtered, 3-seed mean. No single strategy dominates both axes — the Pareto frontier spans multiple strategies depending on budget and metric priority.

Dream-7B-Base PPL vs MAUVE Pareto frontier with native entropy strategy

Figure 10Dream-7B-Base PPL–MAUVE Pareto Frontier

Dream-7B-Base with its native entropy strategy and maskgit. The best strategy changes at every budget, confirming that metric and budget dependence holds across independent model families — not just LLaDA. Origin (no remasking) achieves best MAUVE at 256 steps.

Frontier overlay: PPL-MAUVE scatter and MAUVE bars by strategy and temperature at 256 steps

Figure 5Frontier Overlay — 256 Steps, LLaDA-8B-Base

Left panel: PPL × MAUVE scatter showing strategy × temperature trade-offs at 256 steps. Right panel: MAUVE bars by strategy × temperature confirming the stochasticity-dominated effect. Temperature (t: 0.0→0.25) drives a larger MAUVE shift than any remasking strategy choice — visible in both panels simultaneously.

3 Stochasticity & Diversity

MAUVE vs actual NFE across four stochasticity levels for none, random, high_entropy

Figure 3MAUVE vs Actual NFE Across Stochasticity Levels

LLaDA-8B-Base, English-filtered, 3-seed mean. Four panels: t=0.0, 0.1, 0.25, 0.5. At t=0.0, none dominates across all budgets. At t=0.25, high_entropy collapses to MAUVE=0.652 despite using 3.4× more compute than none (437 vs 128 NFE). The x-axis uses actual NFE — not nominal steps.

Stochasticity interaction: MAUVE and PPL panels for deterministic vs stochastic

Figure 6Stochasticity Interaction — MAUVE & PPL

LLaDA-8B-Base, English-filtered, 3-seed mean. Det = t=0.0; Stoch = t=0.25. MAUVE improves dramatically with stochasticity for none (+0.285 at 128 steps) but high_entropy collapses (−0.151). PPL degrades for all strategies under stochasticity — the two metrics tell opposite stories. Three-way ANOVA: temperature η²=0.908 dominates; Strategy × Temp interaction p=0.002, η²=0.469.

Self-BLEU and Distinct-3 at 256 steps across three strategies and four temperatures

Figure 8Diversity at 256 Steps — Self-BLEU & Distinct-3

LLaDA-8B-Base, English-filtered, 3-seed mean. Self-BLEU (lower = more pairwise-diverse); Distinct-3 (higher = richer vocabulary). high_entropy achieves the lowest Self-BLEU at all temperatures — yet its Distinct-3 collapses at t≥0.25 (0.770 vs none=0.940). Pairwise diversity and vocabulary richness are orthogonal metrics; reporting one without the other is misleading.

4 Token-Level Dynamics

Token stability heatmap: none vs high_entropy, 128 steps, t=0.25, sample 1

Figure 4Token Stability Heatmap

LLaDA-8B-Base, 128 steps, t=0.25, sample 1. Rows = token positions (128); columns = decoding steps (128). Dark = stable (locked) token; light = subsequently remasked. none: 8,129/16,384 stable cells (49.6%). high_entropy: only 2,522/16,384 (15.4%) — persistent token churn causing quality collapse. Full per-cell tensor available in supplementary code archive (heatmap_data.npz).

Token replacement rate per step: none stays at 0%, high_entropy activates at step 40

Figure 7Token Replacement Rate Per Step

t=0.25, 128 steps, 10 samples. none replaces 0% of tokens at every step. high_entropy activates remasking around step 40 (mean replacement 4.3% per step; peak 13.7% at step 41; 88 of 128 steps with non-zero replacement). The late activation followed by persistent churn is the causal mechanism behind MAUVE collapse. Full per-step series in supplementary archive (replacement_rate_data.csv).

Polar area charts: strategy profiles across MAUVE, Fluency, Diversity, Vocab, Efficiency for deterministic and stochastic

Figure 9Polar Area Charts — Deterministic vs Stochastic Strategy Profiles

LLaDA-8B-Base, 128 steps, 3-seed mean. All axes normalized to [0,1] (higher = better). Fluency = 1/PPL rescaled; Diversity = 1 − Self-BLEU; Efficiency = 1/NFE rescaled. (a) Deterministic t=0.0: none dominates on Fluency and Efficiency; strategies differ mainly on fluency. (b) Stochastic t=0.25: none gains large MAUVE but loses Fluency; high_entropy's Vocab axis collapses — confirming that no single strategy wins all axes simultaneously, which is the central motivation for multi-metric evaluation.

Compute-Aware Remasking Evaluation for MDLMs

Leaderboard

Pareto Explorer

Stochasticity Lab

Compute Audit

Model Deep-Dive

CARE Evaluation Protocol

Submit Your Model

Paper Figures