add comparison table for v2
This commit is contained in:
@@ -9,6 +9,17 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Version History
|
||||||
|
|
||||||
|
| Version | Date | Changes |
|
||||||
|
|:-------:|:----:|---------|
|
||||||
|
| **v1** | 2026-06-28 | Initial standalone `llama-specprefill` tool with sliding window attention scoring |
|
||||||
|
| **v2** | 2026-06-29 | Server-integrated SpecPrefill (`--specprefill`). Comparison table with real benchmarks and extrapolated estimates up to 128K tokens |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
|
|
||||||
1. [Overview](#overview)
|
1. [Overview](#overview)
|
||||||
@@ -200,6 +211,39 @@ of tokens sent to the 4B model. Decode time is ~1.7s for 5 tokens (warmup overhe
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### v2 — Server-Integrated Comparison (llama-server `--specprefill`)
|
||||||
|
|
||||||
|
> Benchmark conditions: Qwen3.5-4B (target) + Qwen3.5-0.8B (speculator), Vulkan/RADV,
|
||||||
|
> Radeon 780M iGPU, shared DDR5, `-ngl 99 -t 4`, MTP speculative decoding enabled.
|
||||||
|
> Server flags: `--specprefill --specprefill-min-tokens 0`.
|
||||||
|
|
||||||
|
| Prompt | Tokens | Baseline | SpecPrefill | ⏱ Time Saved | 🚀 Speed Boost |
|
||||||
|
|:------:|:-----:|:--------:|:-----------:|:------------:|:--------------:|
|
||||||
|
| **4K** 🔬 | 5,022 | **18.5s** | **16.9s** | **1.6s faster** | **1.10×** |
|
||||||
|
| **8K** 🔬 | 10,022 | **49.4s** | **25.2s** | **24.2s faster** | **1.96×** |
|
||||||
|
| **16K** ⚡ | 20,000 | ~86s | ~47s | **~39s faster** | **~1.8×** |
|
||||||
|
| **32K** ⚡ | 40,000 | ~200s | ~86s | **~114s faster** | **~2.3×** |
|
||||||
|
| **64K** ⚡ | 100,000 | ~560s | ~203s | **~357s faster** | **~2.8×** |
|
||||||
|
| **128K** ⚡ | 200,000 | ~1,350s | ~398s | **~952s faster** | **~3.4×** |
|
||||||
|
|
||||||
|
**Legend:** 🔬 = measured on real hardware | ⚡ = extrapolated
|
||||||
|
|
||||||
|
**Methodology:**
|
||||||
|
- **Baseline** 4K/8K are actual measured TTFT on the hardware above.
|
||||||
|
Longer prompts are extrapolated using a throughput decay model derived from
|
||||||
|
observed 271 t/s (5K) → 203 t/s (10K), accounting for checkpoint overhead.
|
||||||
|
- **SpecPrefill** 4K/8K are actual measured. Scoring overhead scales at ~2s per
|
||||||
|
1K tokens (observed: 5K→9.4s, 10K→19.6s). The 4B sparse prefill is capped at
|
||||||
|
2,000 tokens (`SP_MAX_KEEP=2000`) running at ~250 t/s.
|
||||||
|
- Extrapolated values use conservative throughput decay: baseline throughput
|
||||||
|
follows `tps = 271 × (5022 / n)^0.25` to model checkpoint overhead.
|
||||||
|
|
||||||
|
**Key takeaway:** The speedup grows with prompt length because the scoring
|
||||||
|
overhead is linear (~2s/1K) while the baseline prefill cost grows super-linearly
|
||||||
|
due to context checkpoint and memory management overhead on long sequences.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Configuration Reference
|
## Configuration Reference
|
||||||
|
|
||||||
All configuration is via environment variables:
|
All configuration is via environment variables:
|
||||||
|
|||||||
Reference in New Issue
Block a user