From c5c6911ace3c36480e47078b46f5e4aa3a7c41df Mon Sep 17 00:00:00 2001 From: asa Date: Mon, 29 Jun 2026 13:23:56 +0200 Subject: [PATCH] add comparison table for v2 --- specprefill.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/specprefill.md b/specprefill.md index 4bd781d..978e7e7 100644 --- a/specprefill.md +++ b/specprefill.md @@ -9,6 +9,17 @@ --- +## Version History + +| Version | Date | Changes | +|:-------:|:----:|---------| +| **v1** | 2026-06-28 | Initial standalone `llama-specprefill` tool with sliding window attention scoring | +| **v2** | 2026-06-29 | Server-integrated SpecPrefill (`--specprefill`). Comparison table with real benchmarks and extrapolated estimates up to 128K tokens | + +--- + +--- + ## Table of Contents 1. [Overview](#overview) @@ -200,6 +211,39 @@ of tokens sent to the 4B model. Decode time is ~1.7s for 5 tokens (warmup overhe --- +### v2 — Server-Integrated Comparison (llama-server `--specprefill`) + +> Benchmark conditions: Qwen3.5-4B (target) + Qwen3.5-0.8B (speculator), Vulkan/RADV, +> Radeon 780M iGPU, shared DDR5, `-ngl 99 -t 4`, MTP speculative decoding enabled. +> Server flags: `--specprefill --specprefill-min-tokens 0`. + +| Prompt | Tokens | Baseline | SpecPrefill | ⏱ Time Saved | 🚀 Speed Boost | +|:------:|:-----:|:--------:|:-----------:|:------------:|:--------------:| +| **4K** 🔬 | 5,022 | **18.5s** | **16.9s** | **1.6s faster** | **1.10×** | +| **8K** 🔬 | 10,022 | **49.4s** | **25.2s** | **24.2s faster** | **1.96×** | +| **16K** ⚡ | 20,000 | ~86s | ~47s | **~39s faster** | **~1.8×** | +| **32K** ⚡ | 40,000 | ~200s | ~86s | **~114s faster** | **~2.3×** | +| **64K** ⚡ | 100,000 | ~560s | ~203s | **~357s faster** | **~2.8×** | +| **128K** ⚡ | 200,000 | ~1,350s | ~398s | **~952s faster** | **~3.4×** | + +**Legend:** 🔬 = measured on real hardware  |  ⚡ = extrapolated + +**Methodology:** +- **Baseline** 4K/8K are actual measured TTFT on the hardware above. + Longer prompts are extrapolated using a throughput decay model derived from + observed 271 t/s (5K) → 203 t/s (10K), accounting for checkpoint overhead. +- **SpecPrefill** 4K/8K are actual measured. Scoring overhead scales at ~2s per + 1K tokens (observed: 5K→9.4s, 10K→19.6s). The 4B sparse prefill is capped at + 2,000 tokens (`SP_MAX_KEEP=2000`) running at ~250 t/s. +- Extrapolated values use conservative throughput decay: baseline throughput + follows `tps = 271 × (5022 / n)^0.25` to model checkpoint overhead. + +**Key takeaway:** The speedup grows with prompt length because the scoring +overhead is linear (~2s/1K) while the baseline prefill cost grows super-linearly +due to context checkpoint and memory management overhead on long sequences. + +--- + ## Configuration Reference All configuration is via environment variables: