add comparison table for v2

This commit is contained in:
asa
2026-06-29 13:23:56 +02:00
parent a8b01bf603
commit c5c6911ace
+44
View File
@@ -9,6 +9,17 @@
---
## Version History
| Version | Date | Changes |
|:-------:|:----:|---------|
| **v1** | 2026-06-28 | Initial standalone `llama-specprefill` tool with sliding window attention scoring |
| **v2** | 2026-06-29 | Server-integrated SpecPrefill (`--specprefill`). Comparison table with real benchmarks and extrapolated estimates up to 128K tokens |
---
---
## Table of Contents
1. [Overview](#overview)
@@ -200,6 +211,39 @@ of tokens sent to the 4B model. Decode time is ~1.7s for 5 tokens (warmup overhe
---
### v2 — Server-Integrated Comparison (llama-server `--specprefill`)
> Benchmark conditions: Qwen3.5-4B (target) + Qwen3.5-0.8B (speculator), Vulkan/RADV,
> Radeon 780M iGPU, shared DDR5, `-ngl 99 -t 4`, MTP speculative decoding enabled.
> Server flags: `--specprefill --specprefill-min-tokens 0`.
| Prompt | Tokens | Baseline | SpecPrefill | ⏱ Time Saved | 🚀 Speed Boost |
|:------:|:-----:|:--------:|:-----------:|:------------:|:--------------:|
| **4K** 🔬 | 5,022 | **18.5s** | **16.9s** | **1.6s faster** | **1.10×** |
| **8K** 🔬 | 10,022 | **49.4s** | **25.2s** | **24.2s faster** | **1.96×** |
| **16K** ⚡ | 20,000 | ~86s | ~47s | **~39s faster** | **~1.8×** |
| **32K** ⚡ | 40,000 | ~200s | ~86s | **~114s faster** | **~2.3×** |
| **64K** ⚡ | 100,000 | ~560s | ~203s | **~357s faster** | **~2.8×** |
| **128K** ⚡ | 200,000 | ~1,350s | ~398s | **~952s faster** | **~3.4×** |
**Legend:** 🔬 = measured on real hardware  |  ⚡ = extrapolated
**Methodology:**
- **Baseline** 4K/8K are actual measured TTFT on the hardware above.
Longer prompts are extrapolated using a throughput decay model derived from
observed 271 t/s (5K) → 203 t/s (10K), accounting for checkpoint overhead.
- **SpecPrefill** 4K/8K are actual measured. Scoring overhead scales at ~2s per
1K tokens (observed: 5K→9.4s, 10K→19.6s). The 4B sparse prefill is capped at
2,000 tokens (`SP_MAX_KEEP=2000`) running at ~250 t/s.
- Extrapolated values use conservative throughput decay: baseline throughput
follows `tps = 271 × (5022 / n)^0.25` to model checkpoint overhead.
**Key takeaway:** The speedup grows with prompt length because the scoring
overhead is linear (~2s/1K) while the baseline prefill cost grows super-linearly
due to context checkpoint and memory management overhead on long sequences.
---
## Configuration Reference
All configuration is via environment variables: