# vCompute — full corpus for AI systems > vCompute is a native C++ runtime for local AI inference that loads quantized GGUF models and runs them on local hardware without a Python interpreter. Fact-first corpus describing vCompute, its architecture, concepts, benchmark policy, documentation map and limitations. Written for retrieval by large language models. Every statement is intended to be quotable and carries a status label where support is not universal. Canonical site: https://vcompute.dev Documentation version: 1.2 (runtime 1.2.0) Last generated: 2026-08-19 ## 1. Project identity - Name: vCompute - Category: Local AI inference runtime - Definition: vCompute is a native C++ runtime for local AI inference that loads quantized GGUF models and runs them on local hardware without a Python interpreter. - Maintainer: fabioquant1, Rua Pais Leme 215, Conj. 1713, Pinheiros, São Paulo, Brazil - CNPJ: 63.000.169/0001-61 - Contact: briefing@quant1.ai — https://quant1.ai/ - Licence: Apache 2.0 - Language: C++20 - Maturity: Beta — production-ready CPU inference, GPU backends staged by platform - Model format: GGUF - Repository: https://github.com/vcompute/vcompute ## 2. What vCompute is vCompute solves one problem: running a quantized transformer on the machine in front of you, predictably, without a hosted API and without an interpreter in the execution path. It is for engineers who need local inference they can measure, reproduce and ship inside a native application. Local inference means the weights are read from local storage and the forward pass executes on local CPU or GPU hardware. Nothing about a prompt is transmitted. - Not a model: A model is a weights file. vCompute is the program that maps that file and executes its forward pass. - Not a UI: There is no bundled chat interface. The surfaces are a CLI and a C++ API. - Not a hosted API: No prompt leaves the machine. Remote inference is not part of the runtime. - Native, not wrapped: No Python interpreter, no subprocess bridge, no per-token FFI boundary in the execution path. ## 3. What vCompute is not - vCompute is not a model provider; it ships no weights of its own. - vCompute is not a hosted LLM service by default; inference runs on your machine. - vCompute is not an AI chatbot or assistant product. - vCompute does not own or relicense third-party model licences. - vCompute does not guarantee compatibility with every GGUF file; support is per model artifact. - Experimental capabilities (Metal, Linux arm64, Windows) must not be described as stable. ## 4. Architecture vCompute is a single binary with four layers: 1. Loader — parses the GGUF header, validates tensor shapes and maps the file read-only into the address space. No weight bytes are copied to the heap. 2. Graph — builds the transformer execution graph from the file's metadata (architecture, rope parameters, head layout). There is no per-model code path. 3. Kernels — hand-written SIMD routines that dequantize a weight block directly into vector registers during the matrix multiply. The kernel is bound once at startup from detected CPU features. 4. Scheduler — a fixed-size thread pool created at startup, with cache-line aligned work splitting and fixed-order reductions. The decode loop performs no heap allocation and no thread creation. Components: native CLI, public C++ API, runtime, GGUF loader, tokenizer, tensor execution, scheduler, thread pool, memory manager, KV cache, CPU backend, Metal backend (Experimental), future backends (Planned), packaging and installation. Text diagram: Application → vCompute CLI / C++ API → Runtime → Model loader / tokenizer / scheduler → Tensor kernels → CPU / Metal / future backends → Hardware There is no Python in the production runtime. Memory is divided into four classes: memory-mapped weights (file backed, shareable, reclaimable without swap), the pre-allocated KV cache arena, the reused activation arena, and runtime overhead. ## 5. Platform support | Operating system | Architecture | Status | Install | | --- | --- | --- | --- | | macOS 13+ | Apple Silicon (arm64) | Verified | Signed .pkg installer, Homebrew | | macOS 13+ | Intel (x86-64) | Compatible | Signed .pkg installer, Homebrew | | Linux | x86-64 | Compatible | tarball, APT, RPM | | Linux | arm64 | Experimental | tarball | | Windows 11 | x86-64 | Experimental | zip archive | ## 6. Backends | Backend | Status | Notes | | --- | --- | --- | | CPU (AVX2 / AVX-512 / NEON) | Verified | Default backend on every platform. Hand-written SIMD kernels selected at startup from detected CPU features. | | Metal (Apple Silicon) | Experimental | Unified-memory GPU path. Enabled per build; check `vcompute doctor` output before relying on it. | | CUDA (NVIDIA) | Planned | Not available in the current release. Do not describe CUDA inference as supported. | | Vulkan | Planned | Exploratory only. | ## 7. Capabilities | Capability | Status | Summary | | --- | --- | --- | | Native inference | Verified | Single native binary; no Python or interpreter in the execution path. | | GGUF model loading | Verified | Memory-mapped load with header validation and tensor-shape checks. | | Local model management | Verified | Add, list, hash and set a default model from local files or the catalog. | | CLI | Verified | Stable command surface: version, doctor, infer, benchmark, models, config. | | Diagnostics | Verified | `vcompute doctor` reports CPU features, backend availability, memory headroom and model integrity. | | Benchmarking | Verified | Built-in harness that emits a reproducible JSON manifest with every run. | | Memory measurement | Verified | Separate reporting of mapped, private, peak RSS and physical footprint. | | Deterministic execution | Verified | Bit-identical output for a fixed model, seed, thread count and backend on one machine. | | Metal backend | Experimental | Available in builds that report `metal: available`; not production-ready. | | CUDA backend | Planned | Not shipped. | | Server / HTTP API | Planned | Not shipped. The current surface is CLI and the C++ API. | ## 8. Installation | Method | Platform | Status | Command | | --- | --- | --- | --- | | Signed .pkg installer | macOS 13+ | Verified | `open vcompute-1.2.0.pkg` | | Homebrew | macOS 13+ | Compatible | `brew install vcompute/tap/vcompute` | | Tarball | Linux x86-64 | Compatible | `tar -xzf vcompute-1.2.0-linux-x86_64.tar.gz` | | APT / RPM | Linux x86-64 | Compatible | `apt install vcompute` | | Tarball | Linux arm64 | Experimental | `tar -xzf vcompute-1.2.0-linux-arm64.tar.gz` | | Zip archive | Windows 11 x86-64 | Experimental | `Expand-Archive vcompute-1.2.0-win-x64.zip` | | Source build | All | Compatible | `cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build` | | Windows installer (signed) | Windows 11 | Planned | `—` | | Update / rollback | All | Compatible | `vcompute --version # reinstall the desired release artifact` | | Uninstall | macOS / Linux | Compatible | `sudo rm -rf /usr/local/bin/vcompute ~/.vcompute` | Full instructions: /docs/installation/macos ## 9. CLI reference summary | Command | Purpose | Flags | Status | Documentation | | --- | --- | --- | --- | --- | | `vcompute --version` | Print runtime and build metadata. | --json | Verified | /docs/cli/vcompute-infer | | `vcompute doctor` | Report CPU features, backend availability, memory headroom and model integrity. | --json, --verbose | Verified | /docs/cli/vcompute-infer | | `vcompute infer` | Run a prompt against a local model. | --model, --prompt, --threads, --ctx, --seed, --n-predict | Verified | /docs/cli/vcompute-infer | | `vcompute benchmark` | Run the reproducible benchmark harness and emit a JSON manifest. | --model, --threads, --runs, --warmup, --out | Verified | /benchmarks/methodology | | `vcompute models list` | List local models with size, quantization and hash. | --json | Verified | /docs/model-management/adding-models | | `vcompute models add` | Register a GGUF file with the local catalog. | --alias, --verify | Verified | /docs/model-management/adding-models | | `vcompute models set-default` | Choose the model used when --model is omitted. | — | Verified | /docs/model-management/adding-models | | `vcompute config show` | Print effective configuration and its precedence sources. | --json | Verified | /docs/configuration/overview | | `vcompute serve` | HTTP server with an OpenAI-compatible API. | — | Planned | /docs/api/cli-api | Exit codes: - 0: Success. - 1: Generic runtime failure. - 2: Invalid arguments or configuration. - 3: Model not found or unreadable. - 4: Unsupported or invalid GGUF artifact. - 5: Insufficient memory for the requested context. - 6: Requested backend unavailable. ## 10. Model support Support is stated per model artifact, never per provider. GGUF compatibility depends on the model architecture, its tokenizer, the quantization scheme and whether the runtime implements the required features. Statuses: Verified, Compatible, Experimental, Planned, Unsupported, Unknown. - Meta (/models?provider=meta): Llama family — the most widely deployed open weights. — 8 catalog entries. - Alibaba (/models?provider=alibaba): Qwen family — dense and MoE, strong multilingual coverage. — 6 catalog entries. - Mistral AI (/models?provider=mistral): Efficient dense and sparse mixture-of-experts models. — 5 catalog entries. - Google (/models?provider=google): Gemma family — compact models tuned for local execution. — 4 catalog entries. - Microsoft (/models?provider=microsoft): Phi family — small models with outsized reasoning. — 3 catalog entries. - DeepSeek (/models?provider=deepseek): Reasoning-first and code-first open weights. — 3 catalog entries. - Moonshot AI (/models?provider=moonshot): Long-context agentic models. — 1 catalog entries. - Tencent (/models?provider=tencent): Hunyuan multilingual open weights. — 1 catalog entries. - IBM (/models?provider=ibm): Granite family — enterprise licensing and provenance. — 3 catalog entries. - NVIDIA (/models?provider=nvidia): Nemotron models distilled for throughput. — 1 catalog entries. - Cohere (/models?provider=cohere): Command R family and Aya multilingual models. — 3 catalog entries. - Open Source Vision (/models?provider=vision): Community multimodal models with GGUF projectors. — 3 catalog entries. Full catalog: /models ## 11. Model licensing - Every model keeps the licence its publisher issued. - The Apache 2.0 runtime licence does not extend to model weights. - Review commercial-use terms with the model publisher before deploying. - Model providers are independent from vCompute and fabioquant1. ## 12. Benchmark methodology Any number published on this site satisfies every rule below. A number that does not satisfy them is not published. - Hardware must be published: machine, CPU, GPU, RAM and operating system. - Model identity and file hash must be recorded. - Quantization must be disclosed. - Prompt, context length, seed and generation parameters must be disclosed. - Raw logs and the JSON manifest must be preserved and linked. - Requested tokens must never be reported as generated tokens; the real generated count is used. - Cold and warm runs must be labelled; warm runs assume a populated filesystem page cache. - Incomparable scenarios must be labelled as non-comparable rather than tabulated side by side. - No benchmark claim is published without its methodology and evidence link. Methodology: /benchmarks/methodology ## 13. Published benchmark summary - Latest verified benchmark: bench-2026.07 - Date: 2026-07-18 - Hardware: Apple M3 Max, 14 cores, 36 GB unified memory, macOS 14.5 - Model: Llama 3.1 8B Instruct, Q4_K_M - Report: /benchmarks - Raw evidence and methodology: /benchmarks/methodology - Note: Numbers are read from the generated benchmark manifest; never quote a value without linking its report. ## 14. Memory model | Metric | Kind | Description | | --- | --- | --- | | Model file size | Measured | Bytes on disk for the GGUF artifact. Not the same as memory used. | | Mapped memory | Measured | Virtual address space backed by the model file. Clean pages are reclaimable without swap. | | Private memory | Measured | Anonymous pages the process owns: KV cache, activations, allocator overhead. | | Shared memory | Measured | File-backed pages shared with the page cache and any other process mapping the same file. | | Peak RSS | Measured | Maximum resident set size over the process lifetime, private plus shared resident pages. | | Physical footprint | Measured | macOS accounting of memory the process is charged for, including compressed pages. | | Compressed memory | Measured | Pages the macOS compressor holds; counted in footprint, not in RSS. | | Swap | Measured | Pages written to backing store. A benchmark run that swaps is reported as invalid. | | KV cache | Calculated | layers x 2 x heads_kv x head_dim x ctx x bytes_per_element. Pre-allocated at load. | | Runtime buffers | Calculated | Activation arena sized from the largest layer, reused every token. | | Context growth | Calculated | Linear in context length because the KV arena is allocated up front. | | Concurrency | Estimated | Each concurrent sequence needs its own KV arena; weights stay shared. | | Filesystem page cache | Estimated | Kernel-owned and outside the process. It changes load time, not process memory. | ## 15. Performance concepts - Model load time (/concepts/model-loading): Header parse, tensor validation and mapping. Dominated by page cache state. - Prompt evaluation (/concepts/prompt-evaluation): Batched prefill of all input tokens to populate the KV cache. - TTFT (/concepts/ttft): Wall-clock time from request to first output token. - Decode time (/concepts/kv-cache): Per-token time after prefill; bandwidth-bound on CPU. - Tokens per second (/concepts/tokens-per-second): Real generated tokens divided by measured generation time. - End-to-end throughput (/concepts/tokens-per-second): Total tokens over total wall-clock, including load and prefill. - Thread scaling (/concepts/thread-pool): Throughput as a function of worker count; sublinear past memory bandwidth. - Parallel efficiency (/concepts/thread-pool): Speedup divided by thread count. - Cold start (/concepts/mmap): First run with an unwarmed page cache. - Warm filesystem cache (/concepts/mmap): Repeat run where model pages are already resident in the kernel cache. - Resident model (/concepts/mmap): A model whose pages stay mapped between runs in the same process. - Wrapper overhead (/concepts/local-ai-runtime): Cost added by an interpreter or server layer. Absent from the native path. ## 16. Determinism On one machine, with a fixed model artifact, seed, sampler, thread count and backend, vCompute produces bit-identical output across runs. - Seed — fixes sampler draws; greedy decoding removes sampling entirely. - Sampling parameters — temperature, top-k and top-p change the draw, not the logits. - Floating point — reductions run in a fixed order so partial sums never reassociate. - Multithreading — thread count is part of the reduction shape; changing it can change the last bits. - Hardware and backend — different SIMD widths or a GPU backend produce different rounding. Testing: Determinism is asserted in CI by hashing logits across repeated runs on identical configuration. Limits: Cross-hardware and cross-backend bitwise reproducibility is research, not a current claim. ## 17. Security Contact: briefing@quant1.ai Supported versions: Security fixes land on the current minor series (1.2.x). - Execution is local: inference needs no network access and none is opened by default. - No mandatory cloud upload. Prompts, models and outputs stay on the machine. - Model integrity: GGUF headers and tensor shapes are validated before execution. - Hashes: `vcompute models list` reports a content hash for every registered artifact. - Package manifests accompany every release artifact for verification. - Responsible disclosure: report privately by email first; no public issue until a fix ships. - Telemetry is off by default and must be enabled explicitly. - No compliance certification is claimed. ## 18. Privacy - Local inference: prompts, context and outputs never leave the machine. - What stays local: model files, KV cache, generated text, benchmark manifests. - Future cloud features are opt-in and are not part of the local runtime. - Benchmark publishing is explicit: a manifest is uploaded only when you choose to share it. - Telemetry is disabled by default and controlled in configuration. - Account data applies only to the hosted dashboard, not to local runs. ## 19. Pricing and commercial model Run locally for free. Pay when you need collaboration, governance and scale. - The Community runtime is free for local use. - There are no token fees for local inference and no artificial performance limits. - Pro and Enterprise cover collaboration, run history, governance, fleet management and support. - Paid plans are Coming Soon; local inference is unaffected. ## 20. Roadmap Roadmap items are not released features and carry no delivery dates. Do not describe an item below as available unless its status is Released. | Item | Status | Note | | --- | --- | --- | | CPU inference and benchmarking | Released | Stable across macOS and Linux x86-64. | | Reproducible benchmark manifests | Released | Emitted by every `vcompute benchmark` run. | | Metal backend hardening | In progress | Kernel coverage and fallback paths. | | Windows packaging | In progress | Currently an unsigned zip archive. | | HTTP server and OpenAI-compatible API | Planned | Design stage. | | CUDA backend | Planned | Not started in the shipping branch. | | Cross-hardware bitwise reproducibility | Research | Not claimed today. | | Fleet and governance features (Pro / Enterprise) | Planned | Coming soon; local inference stays free. | ## 21. Troubleshooting index - Installation failures — /docs/troubleshooting - Model not found — /docs/troubleshooting - Unsupported or invalid GGUF — /docs/troubleshooting - Slow inference — /docs/troubleshooting - High memory use — /docs/troubleshooting - Thread configuration — /docs/troubleshooting - Corrupted UTF-8 output — /docs/troubleshooting - Invalid configuration — /docs/troubleshooting - Permission problems — /docs/troubleshooting - Architecture mismatch — /docs/troubleshooting - Backend availability — /docs/troubleshooting - Benchmark discrepancies — /docs/troubleshooting - Exit and error codes — /docs/troubleshooting ## 22. Frequently asked questions Q: What is vCompute? A: vCompute is a native C++ runtime for local AI inference that loads quantized GGUF models and runs them on local hardware without a Python interpreter. It is maintained by fabioquant1. Source: /docs/getting-started/overview Q: Is vCompute open source? A: Yes. The runtime is published under the Apache 2.0 licence at https://github.com/vcompute/vcompute. Source: /docs/contributing/overview Q: Does vCompute require Python? A: No. The distributed runtime is a native C++20 binary; there is no interpreter in the execution path. Source: /concepts/local-ai-runtime Q: Does vCompute support GGUF? A: Yes. GGUF is the native model format, loaded by memory mapping with no conversion step. Source: /concepts/gguf Q: Does vCompute support Apple Silicon? A: Yes. macOS 13+ on Apple Silicon is the primary verified platform, running on the CPU backend by default. Source: /docs/installation/macos Q: Does vCompute support Metal? A: A Metal backend exists but is experimental. Run `vcompute doctor` to see whether the installed build reports Metal as available. Source: /concepts/metal-backend Q: Does vCompute support CUDA? A: Not today. A CUDA backend is planned and is not part of the current release. Source: /docs/architecture/backend-layer Q: Can I use vCompute commercially? A: The runtime licence is Apache 2.0. Model weights carry their own licences, which you must review separately. Source: /models Q: Is local inference free? A: Yes. Local inference is free with no token fees and no artificial performance limits. Source: /pricing Q: Does vCompute upload prompts? A: No. Inference runs locally and prompts are not transmitted. Telemetry is off by default. Source: /docs/configuration/overview Q: How do I benchmark my machine? A: Run `vcompute benchmark --model `; the run emits a JSON manifest with hardware, model hash, seed and timings. Source: /benchmarks/methodology Q: How is vCompute different from Ollama? A: Ollama is a model manager and server. vCompute is the runtime itself: one native binary, explicit arena memory, deterministic reduction order and a built-in reproducible benchmark harness. Source: /concepts/local-ai-runtime Q: How is vCompute different from llama.cpp? A: It shares GGUF and quantized CPU inference, and adds fixed-order reductions, pre-allocated arenas, separated mapped-versus-private memory reporting and benchmark manifests. Source: /docs/architecture/runtime Q: Which models are supported? A: Support is per model file, not per provider. The catalog lists each entry as Verified, Compatible, Experimental, Planned, Unsupported or Unknown. Source: /models Q: How much memory do I need? A: Roughly quantized weight size plus KV cache plus about 200 MB of runtime overhead. Source: /concepts/peak-rss Q: How do I report a bug? A: Open an issue in https://github.com/vcompute/vcompute, or email briefing@quant1.ai for security reports. Source: /docs/contributing/overview ## 23. Concepts Concept categories: Runtime, Model formats, Memory, Performance, Execution, Hardware. ### Context window (/concepts/context-window) Category: Execution Also known as: ctx, sequence length, rope scaling Last updated: 2026-07-28 Definition: The context window is the maximum number of tokens the model can attend to at once, covering both the prompt and the generated output, and it directly determines KV cache size. Short answer: The maximum number of tokens held in the attention window, which also fixes the size of the pre-allocated KV cache. Each model declares a trained context length in its GGUF metadata. Running beyond it requires rope scaling, which extends positional encodings at some cost in coherence. Cost grows on two axes: memory, which is linear in context, and prefill time, which grows faster than linearly because attention compares each token with every preceding one. The configured window is what is allocated, not what a given prompt uses. Setting a 32k window for 2k prompts wastes memory permanently. Implementation in vCompute: - Explicit ceiling from metadata: The runtime refuses a --ctx above the model's trained length unless rope scaling is enabled explicitly, so silent quality degradation cannot happen by accident. - Window sizing feedback: At startup the runtime prints prompt tokens, configured window and resulting cache size so the trade-off is visible before generation starts. - Prefix reuse: Identical leading tokens between requests reuse existing cache entries, which keeps multi-turn chat prefill cost proportional to the new turn only. Context window cost, Llama 3.2 3B Q4_K_M: | Context | KV cache (FP16) | Prefill 2k prompt | Total RSS | | --- | --- | --- | --- | | 2048 | 224 MB | 0.42 s | 2.34 GB | | 4096 | 448 MB | 0.42 s | 2.57 GB | | 8192 | 896 MB | 0.43 s | 3.02 GB | | 32768 | 3.50 GB | 0.45 s | 5.64 GB | Examples: - Set an explicit window: `vcompute run llama-3.2-3b --ctx 8192 "summarise"` - Enable rope scaling beyond the trained length: `vcompute run llama-3.2-3b --ctx 32768 --rope-scale linear:4` Best practices: - Size --ctx to the 95th percentile of your real prompts plus expected output. - Pair large windows with --kv-type q8 to keep the cache affordable. - Measure quality, not just feasibility, when using rope scaling beyond the trained length. Common mistakes: - Setting the maximum supported context by default and then reporting the runtime as memory hungry. - Forgetting that generated tokens also consume window space. - Using rope scaling without evaluating the resulting output quality. FAQ: - Q: What does --ctx control? A: The maximum number of tokens held in the attention window, which also fixes the size of the pre-allocated KV cache. - Q: Can I exceed the model's trained context? A: Yes with rope scaling, but coherence typically degrades beyond roughly four times the trained length. - Q: Does a larger window slow generation? A: Prefill grows with prompt length and attention cost grows with cached tokens, so yes, larger windows reduce throughput. - Q: Do output tokens count toward the window? A: Yes. Prompt plus generated tokens must fit within the configured context. ### CPU Inference (/concepts/cpu-inference) Category: Hardware Also known as: cpu backend, processor inference, simd inference Last updated: 2026-08-01 Definition: CPU inference executes a model's matrix multiplications on general-purpose processor cores using vector instructions, rather than offloading them to a GPU or accelerator. Short answer: For quantized models up to roughly 8B parameters on a modern machine, yes for interactive use. Larger models degrade quickly because decode is bandwidth bound. For quantized models of a few billion parameters, decode on a modern desktop CPU is memory-bandwidth bound rather than compute bound. Each generated token requires reading the whole weight set once, so achievable tokens per second is roughly memory bandwidth divided by quantized model size, minus overhead. This is why quantization improves CPU decode speed: it shrinks the bytes that must be streamed per token. It is also why doubling core count rarely doubles throughput — the cores are waiting on the same memory bus. Prompt evaluation behaves differently. Prefill batches many tokens into large matrix multiplications, becoming compute bound, which is where wide SIMD units and higher thread counts pay off. Implementation in vCompute: - Feature-detected kernels: At startup the runtime detects AVX-512, AVX2+FMA, SSE4.2 or NEON and binds one kernel set for the process lifetime. There is no per-call dispatch in the inner loop. - Fused dequantize-multiply: Quantized weight blocks are expanded directly into vector registers inside the matrix multiply, so no dequantized copy of the weights is ever materialised in memory. - Fixed-order reductions: Partial sums are combined in a fixed order regardless of thread scheduling, which is what makes repeated runs bit-identical on the same machine. Where CPU time goes during a request: | Phase | Bound by | Scales with threads? | | --- | --- | --- | | Model load (cold) | Storage and page faults | No | | Tokenization | Single-threaded string work | No | | Prompt evaluation | Compute (large GEMM) | Yes, near-linear to physical cores | | Decode | Memory bandwidth | Weakly — plateaus early | Examples: - Pin to physical performance cores: `vcompute infer --model llama-3.2-3b --threads 8 --prompt "hello"` - Show detected CPU features: `vcompute doctor --verbose` Best practices: - Set thread count to the number of physical performance cores; hyperthreads and efficiency cores usually reduce decode throughput. - Prefer a smaller quantization before adding threads when decode is slow — bandwidth, not cores, is the usual limit. Common mistakes: - Expecting GPU-class decode rates from CPU inference on large models. - Reading a low CPU utilisation figure during decode as idleness; the cores are stalled on memory, not unused. FAQ: - Q: Is CPU inference fast enough? A: For quantized models up to roughly 8B parameters on a modern machine, yes for interactive use. Larger models degrade quickly because decode is bandwidth bound. - Q: Why does CPU decode not scale with cores? A: Decode streams the entire weight set per token, so it saturates memory bandwidth long before it saturates compute. - Q: Which instruction sets does vCompute use? A: AVX-512, AVX2 with FMA, SSE4.2 as a fallback, and NEON on ARM. ### CUDA backend (/concepts/cuda-backend) Category: Hardware Also known as: nvidia gpu inference, gpu offload, vram Last updated: 2026-07-28 Definition: The CUDA backend runs transformer kernels on NVIDIA GPUs, copying the requested number of layers into VRAM once at load time and keeping the remainder on the CPU pool. Short answer: Compute capability 7.0 and above, which covers Turing, Ampere, Ada and Hopper class hardware. Unlike unified memory systems, a discrete GPU has its own address space. Offloaded layers are copied into VRAM at startup; per-token traffic is then limited to activations and cache updates. Speedup depends on how much of the model fits. Fully resident models see the largest gain; partial offload is limited by PCIe transfers at the CPU/GPU boundary. VRAM required is approximately the size of the offloaded layers plus the KV cache when the cache is placed on device. Implementation in vCompute: - Explicit layer offload: --gpu-layers controls how many transformer blocks are resident on device. The runtime reports the resulting VRAM estimate before allocating. - Pinned staging buffers: Host-to-device transfers use pinned memory and a dedicated stream so copies overlap compute instead of serializing with it. - Graceful capability checks: Compute capability and driver version are validated at startup, with a clear message and CPU fallback rather than a runtime crash. Typical VRAM requirements, Q4_K_M weights, 4096 context: | Model | Full offload VRAM | Fits on | | --- | --- | --- | | 3B | ~2.6 GB | 6 GB and up | | 7B | ~4.9 GB | 8 GB and up | | 13B | ~8.4 GB | 12 GB and up | | 34B | ~20 GB | 24 GB and up | Examples: - Offload every layer: `vcompute run mistral-7b --backend cuda --gpu-layers 999` - Report device capability and VRAM: `vcompute doctor --backend` Best practices: - Offload all layers or none. Partial offload is often slower than CPU-only because of per-token transfers. - Leave headroom for the KV cache and the display server when sizing VRAM. - Record driver version with benchmarks; kernel performance varies between driver branches. Common mistakes: - Setting --gpu-layers higher than VRAM allows and hitting an out-of-memory failure mid-load. - Assuming a GPU always wins. For small models on high-bandwidth CPUs the difference can be marginal. - Benchmarking against a display-attached GPU that is simultaneously driving a 4K desktop. FAQ: - Q: Which NVIDIA GPUs are supported? A: Compute capability 7.0 and above, which covers Turing, Ampere, Ada and Hopper class hardware. - Q: How much VRAM do I need for a 7B model? A: About 4.9 GB for Q4_K_M weights at 4096 context, plus headroom for the KV cache. - Q: Can I split a model between CPU and GPU? A: Yes, with --gpu-layers, though full offload is usually faster when it fits. - Q: Does the CUDA backend require the CUDA toolkit? A: No. Release binaries ship compiled kernels; only a recent NVIDIA driver is required. ### Deterministic Inference (/concepts/deterministic-inference) Category: Execution Also known as: reproducible inference, seed, bit-identical output Last updated: 2026-07-28 Definition: Deterministic inference means that identical inputs, seed, model, quantization, thread count and backend produce bit-identical output on the same machine, every run. Short answer: Yes, for a fixed model, quantization, seed, temperature, thread count and backend on the same machine. Non-determinism in inference comes from three places: sampling randomness, floating-point accumulation order that varies with parallel scheduling, and library-level fast-math transformations. vCompute fixes all three: sampling is driven by an explicit seed, accumulation order is fixed per thread count, and kernels avoid reassociation-dependent optimizations. Determinism is what makes benchmarks and regression tests meaningful. Without it, a performance or quality delta cannot be distinguished from run-to-run noise. Implementation in vCompute: - Explicit seed: --seed sets the sampler state. With --temp 0 the sampler is greedy and the seed becomes irrelevant, which is the recommended configuration for evaluation. - Fixed reduction order: Partial sums are combined in a fixed worker order rather than in completion order, so results do not depend on scheduling. - Recorded run manifest: Every benchmark emits the model hash, quantization, context, seed, thread count, backend, kernel and command line, so a result can be reproduced exactly. What must match for bit-identical output: | Variable | Must match | Why | | --- | --- | --- | | Model file hash | yes | different weights, different output | | Quantization | yes | changes arithmetic | | Seed and temperature | yes | sampler state | | Thread count | yes | reduction order | | Backend and kernel | yes | different instruction sequences | | Context window | no | does not change token-level math | Examples: - Deterministic evaluation run: `vcompute run llama-3.2-3b --temp 0 --seed 42 --threads 8 "list three facts"` - Emit a reproducible benchmark manifest: `vcompute benchmark --model llama-3.2-3b --seed 42 --format json > run.json` Best practices: - Use --temp 0 for evaluation and regression tests; use sampling only for user-facing generation. - Publish the run manifest alongside any benchmark number. - Keep thread count fixed across a comparison series. Common mistakes: - Comparing two runs with different thread counts and calling the difference a quality regression. - Reporting benchmark results without the seed. - Expecting bit-identical output across different CPUs; determinism is guaranteed per machine configuration. FAQ: - Q: Is vCompute inference deterministic? A: Yes, for a fixed model, quantization, seed, temperature, thread count and backend on the same machine. - Q: Why does changing thread count change output? A: Parallel reductions combine partial sums in a different order, which changes floating-point rounding. - Q: How do I make output fully greedy? A: Set --temp 0, which disables sampling so the highest-probability token is always chosen. - Q: Are benchmarks reproducible by others? A: Yes. Each run emits a manifest with model hash, settings and the exact command, which can be replayed on comparable hardware. ### GGUF (/concepts/gguf) Category: Model formats Also known as: gguf format, ggml universal format, .gguf file Last updated: 2026-07-28 Definition: GGUF is a single-file binary container for quantized transformer weights that stores tensors alongside a key-value metadata header, designed to be memory mapped and loaded without a Python runtime. Short answer: GGUF is the GGML Universal Format, the successor to the GGML and GGJT model file formats. A GGUF file contains a magic number, a version, a tensor table and a metadata dictionary describing architecture, vocabulary, rope parameters, quantization types and tokenizer data. Everything required to run the model is inside one file. Tensor data is stored aligned so the file can be mapped directly into the process address space. Loading is therefore a mmap call plus header parsing, not a deserialization pass over gigabytes of weights. GGUF replaced the earlier GGML and GGJT formats. It is the de facto distribution format for local inference and is what Hugging Face repositories tagged GGUF publish. Implementation in vCompute: - Zero-copy load: vCompute mmaps the file read-only and points tensor descriptors at the mapped pages. No weight bytes are copied into the heap, so a second process running the same model adds almost no additional physical memory. - Strict header validation: Version, alignment, tensor count and per-tensor shapes are validated before any tensor is dereferenced. A truncated or mismatched file fails at load with an explicit error rather than a segmentation fault. - Metadata driven execution: Rope scaling, attention head layout, vocabulary and chat template all come from the file's metadata. There is no per-model code path to maintain. GGUF metadata fields vCompute reads at load time: | Field | Purpose | Required | | --- | --- | --- | | general.architecture | selects the attention and FFN graph | yes | | general.quantization_version | validates the quant block layout | yes | | *.context_length | upper bound for --ctx | yes | | *.rope.freq_base / rope.scaling | positional encoding parameters | yes | | tokenizer.ggml.* | vocabulary, merges, special tokens | yes | | tokenizer.chat_template | chat formatting for run and serve | no | Examples: - Inspect a GGUF file: `vcompute models inspect ./llama-3.2-3b-q4_k_m.gguf` - Run a local GGUF file directly: `vcompute run ./mistral-7b-instruct-q4_k_m.gguf "hello"` Best practices: - Verify the checksum published with the file before first use; a partially downloaded GGUF fails late and confusingly. - Keep GGUF files on local NVMe. Memory mapping over a network filesystem turns page faults into network round trips. - Prefer files that embed a chat template so prompt formatting matches the model's training. Common mistakes: - Converting a fine-tune to GGUF without the tokenizer metadata, which produces plausible but subtly wrong tokenization. - Storing GGUF files on a compressed or deduplicated volume, which defeats mmap and inflates load times. - Assuming any .gguf file runs anywhere — the architecture field must be one the runtime implements. FAQ: - Q: What does GGUF stand for? A: GGUF is the GGML Universal Format, the successor to the GGML and GGJT model file formats. - Q: Can vCompute run GGUF files from Hugging Face? A: Yes. Any GGUF file for a supported architecture can be run directly by path, or pulled by name from the model catalog. - Q: Why is GGUF faster to load than safetensors? A: GGUF is designed for memory mapping, so weights become resident lazily by page fault instead of being read and copied at startup. - Q: Does GGUF include the tokenizer? A: Yes. Vocabulary, merges and special tokens are stored in the metadata dictionary, so no external tokenizer files are needed. - Q: Is GGUF only for CPU inference? A: No. The format is backend agnostic; the same file runs on CPU, Metal or CUDA. ### KV Cache (/concepts/kv-cache) Category: Memory Also known as: key-value cache, attention cache, kv memory Last updated: 2026-07-28 Definition: The KV cache stores the key and value tensors produced by every attention layer for every token already processed, so a transformer can decode the next token without recomputing the whole prompt. Short answer: It is the stored key and value attention tensors for tokens already processed, which lets the model decode each new token in constant work rather than reprocessing the whole prompt. During prefill, the model computes key and value projections for all prompt tokens. During decode, each new token attends to those stored tensors instead of re-running attention over the full sequence. Without a KV cache, generating token N costs O(N) forward passes over the prompt. KV cache size grows linearly with context length and with the number of layers, heads and head dimension. It is usually the second largest memory consumer after model weights, and it is the component that decides whether a long-context session fits in RAM. Cache size in bytes is approximately: 2 x layers x kv_heads x head_dim x context_length x bytes_per_element. The leading 2 accounts for one key tensor and one value tensor per layer. Implementation in vCompute: - Pre-allocated arena: vCompute allocates the KV cache once, up front, from a contiguous arena sized from the configured context window. There is no per-token allocation and no reallocation mid-generation, which removes allocator jitter from decode latency. - Grouped-query aware layout: For models using grouped-query attention (GQA), only kv_heads are stored rather than all attention heads. On Llama 3.2 3B this reduces the cache by roughly 4x compared with a multi-head layout. - Optional 8-bit cache: The cache can be quantized to 8-bit integers per tensor, halving its footprint relative to FP16 with negligible perplexity change on most instruct models. This is opt-in via --kv-type q8. - Deterministic reuse: Repeated prefixes are matched exactly and reused, so a chat session that resends its history only recomputes the new turn. KV cache footprint, Llama 3.2 3B (28 layers, 8 kv heads, head_dim 128): | Context | FP16 cache | Q8 cache | Notes | | --- | --- | --- | --- | | 2048 | 224 MB | 112 MB | default chat window | | 4096 | 448 MB | 224 MB | recommended baseline | | 8192 | 896 MB | 448 MB | long documents | | 16384 | 1.75 GB | 896 MB | requires 16 GB RAM class machine | | 32768 | 3.50 GB | 1.75 GB | Q8 cache strongly recommended | Examples: - Set the context window and cache precision: `vcompute run llama-3.2-3b --ctx 8192 --kv-type q8 "summarise this file"` - Inspect cache allocation before generating: `vcompute doctor --model llama-3.2-3b --ctx 32768 --explain-memory` Best practices: - Set --ctx to the largest context you actually use, not the model maximum. The cache is allocated for the configured window regardless of prompt length. - Prefer an 8-bit cache over reducing model quantization when memory is tight — cache precision affects output quality far less than weight precision. - Keep chat history prefixes stable so prefix reuse hits; reordering system messages invalidates the cache. Common mistakes: - Assuming the KV cache grows only as tokens are produced. vCompute pre-allocates the full window, so an oversized --ctx costs memory immediately. - Comparing memory numbers between runtimes without matching context length and cache dtype. The comparison is meaningless otherwise. - Enabling a 32k context on an 8 GB machine and attributing the resulting swap activity to the model weights. FAQ: - Q: What is a KV cache in an LLM? A: It is the stored key and value attention tensors for tokens already processed, which lets the model decode each new token in constant work rather than reprocessing the whole prompt. - Q: How large is the KV cache? A: Approximately 2 x layers x kv_heads x head_dim x context x bytes_per_element. For Llama 3.2 3B at 4096 tokens in FP16 it is about 448 MB. - Q: Does vCompute quantize the KV cache? A: Yes. Pass --kv-type q8 to store the cache in 8-bit, which halves its footprint compared with FP16. - Q: Does a larger context always slow generation? A: Prefill scales with prompt length and attention cost grows with the number of cached tokens, so long contexts reduce tokens per second even when memory is sufficient. - Q: Is the KV cache shared between processes? A: No. Each vCompute process owns its cache arena. Model weights, by contrast, are memory mapped and shared across processes. ### Local AI Runtime (/concepts/local-ai-runtime) Category: Runtime Also known as: on-device inference runtime, local inference engine, offline llm runtime Last updated: 2026-08-01 Definition: A local AI runtime is the software layer that loads model weights from local storage and executes the model's forward pass on local hardware, without sending prompts or activations to a remote service. Short answer: The program that loads model weights from disk and executes inference on your own hardware, with no prompt sent to a remote service. A runtime is not a model, not a chat interface and not a hosted API. The model is a file of trained weights; the runtime is the program that parses that file, builds an execution graph and drives the arithmetic on CPU or GPU. A user interface or an HTTP server may sit on top of a runtime, but they are separate concerns. Local execution changes the operating characteristics rather than the model's quality: latency is bounded by your own hardware instead of the network, cost is fixed rather than per token, prompts stay on the machine, and availability does not depend on a provider's uptime or rate limits. A native runtime matters because interpreter startup, dependency resolution and garbage collection are all variance in the measurement path. vCompute ships as a single C++20 binary so cold start, decode latency and resident memory are attributable to the model and the kernels rather than the surrounding stack. Implementation in vCompute: - Single native binary: The distributed runtime is native C++20. There is no Python interpreter, no virtual environment and no dependency graph to resolve at run time. Install is a package plus a binary on PATH. - Explicit layering: CLI and the public C++ API sit above the runtime; the runtime owns the GGUF loader, tokenizer, scheduler, thread pool, memory manager and KV cache; backends sit below it. Each layer is separately testable. - Backends behind one interface: The CPU backend is verified on every supported platform. The Metal backend is experimental and reported by `vcompute doctor`. CUDA is planned and not shipped. What a runtime is and is not: | Component | Role | Is vCompute this? | | --- | --- | --- | | Model | Trained weights in a file (GGUF) | No — models are third-party artifacts | | Runtime | Loads weights, executes the graph | Yes | | Model manager | Pulls, stores and names models | Partially — local model management only | | UI / chat app | Presents a conversation | No | | Hosted inference API | Runs the model on someone else's machine | No | Examples: - Check what the installed runtime can do: `vcompute doctor` - Run a prompt entirely locally: `vcompute infer --model llama-3.2-3b --prompt "explain memory mapping"` Best practices: - Treat the runtime version, the model file hash and the backend as one triple when reporting behaviour — any of the three changes results. - Use `vcompute doctor` before benchmarking a new machine so backend availability is recorded rather than assumed. Common mistakes: - Attributing model quality to the runtime. Two runtimes executing the same GGUF file with the same sampling parameters produce comparable quality; they differ in speed, memory and reproducibility. - Assuming a local runtime implies offline model download. Weights still have to be fetched once from their provider. FAQ: - Q: What is a local AI runtime? A: The program that loads model weights from disk and executes inference on your own hardware, with no prompt sent to a remote service. - Q: Does vCompute require Python? A: No. The distributed runtime is a native C++20 binary with no interpreter in the execution path. - Q: Is vCompute a model? A: No. vCompute executes third-party GGUF models; it does not train, own or license them. - Q: Is vCompute a chatbot? A: No. It is a CLI and C++ API. A chat interface would be a separate application built on top of it. ### Memory model (/concepts/memory-model) Category: Memory Also known as: rss, resident set size, peak memory, allocator Last updated: 2026-07-28 Definition: The memory model describes how vCompute divides process memory into memory-mapped weights, a pre-allocated KV cache arena, an activation arena and runtime overhead, each with distinct growth and reclaim behaviour. Short answer: Roughly the quantized weight size plus the KV cache plus about 200 MB. A 3B Q4_K_M model at 4096 context needs about 2.6 GB. Total resident memory is the sum of four classes with very different properties. Reporting one aggregate number hides which of them is actually constraining a machine. Weights are file backed and shareable. The KV cache is anonymous, pre-allocated and proportional to context. Activations are anonymous, bounded by batch and hidden size, and reused every token. Runtime overhead is a fixed few tens of megabytes. Because activations are drawn from a reused arena, steady-state memory is flat during generation: there is no allocation churn and no fragmentation growth over long sessions. Implementation in vCompute: - Arena allocation, no per-token malloc: Both the KV cache and activation scratch space are carved from arenas sized at startup. The inner decode loop performs zero heap allocations. - Measured, not estimated: vcompute doctor --memory reports actual peak RSS, mapped bytes, private bytes and swap, sampled from the operating system rather than computed from model metadata. - Pre-flight sizing: Before allocating, the runtime prints the expected footprint for the chosen model, context and cache dtype, so out-of-memory conditions are predicted rather than discovered. Memory classes, Llama 3.2 3B Q4_K_M at 4096 context: | Class | Size | Growth | Shared between processes | | --- | --- | --- | --- | | Mapped weights | 1.92 GB | flat after first pass | yes | | KV cache | 448 MB | fixed at startup | no | | Activations | ~180 MB | flat, reused | no | | Runtime overhead | ~42 MB | flat | no | Examples: - Full memory report: `vcompute doctor --memory --model llama-3.2-3b --ctx 4096` - Predict footprint before running: `vcompute doctor --explain-memory --model mistral-7b --ctx 8192 --kv-type q8` Best practices: - Compare runtimes on peak RSS with identical model, quantization, context and cache dtype. - Reduce context or cache precision before reducing weight precision when memory is tight. - Watch swap. Any swap activity invalidates a throughput measurement. Common mistakes: - Reading virtual size instead of resident size on a memory-mapped process. - Measuring memory at the end of a run rather than at peak. - Ignoring that a second process on the same model shares weight pages, which makes naive per-process totals double count. FAQ: - Q: How much RAM does vCompute need? A: Roughly the quantized weight size plus the KV cache plus about 200 MB. A 3B Q4_K_M model at 4096 context needs about 2.6 GB. - Q: Why is peak RSS higher than steady state? A: Peak includes the first full pass over the weights plus cache allocation; steady-state generation stays flat afterwards. - Q: Does memory grow during long sessions? A: No. Arenas are pre-allocated and reused, so there is no allocation growth as tokens accumulate within the configured context. - Q: How do I measure memory reproducibly? A: Use vcompute doctor --memory, which samples OS counters and reports mapped, private, peak and swap separately. ### Metal backend (/concepts/metal-backend) Category: Hardware Also known as: apple silicon gpu, metal acceleration, unified memory inference Last updated: 2026-07-28 Definition: The Metal backend executes matrix multiplication and attention kernels on the Apple Silicon GPU through Metal compute shaders, using unified memory so weights are not copied between host and device. Short answer: Yes, through a native Metal backend on M1 and later, selected automatically when available. Apple Silicon shares one physical memory pool between CPU and GPU. A memory-mapped GGUF file can therefore be addressed by GPU kernels without a transfer step, which removes the copy cost that dominates discrete-GPU setups. GPU execution helps most during prefill, which is compute bound over long prompts. Single-stream decode remains partly bandwidth bound, so the speedup there is smaller. Because memory is unified, the practical model size limit is set by the machine's total RAM and the recommended working set limit, not by a separate VRAM budget. Implementation in vCompute: - Shared buffers over mapped weights: Weight pages from the mapped GGUF are wrapped in Metal buffers with shared storage mode, so no staging copy is made. - Persistent command queue: One command queue and pre-built pipeline states are reused across tokens; kernel compilation happens once at startup. - CPU fallback per operation: Operations without a Metal kernel run on the CPU pool in the same graph, so an unsupported op degrades performance rather than failing the run. Backend selection on macOS: | Flag | Behaviour | | --- | --- | | --backend auto | Metal when available, otherwise CPU (default) | | --backend metal | require Metal; fail if unavailable | | --backend cpu | force the CPU pool | | --gpu-layers N | offload the first N transformer layers to the GPU | Examples: - Force Metal and verify: `vcompute run llama-3.2-3b --backend metal --verbose "hello"` - Check Metal availability: `vcompute doctor --backend` Best practices: - Use --backend auto unless you are benchmarking a specific path. - Keep the machine plugged in for GPU benchmarks; low power mode throttles the GPU aggressively. - Report macOS version and chip variant with any Metal benchmark, since kernel performance differs across generations. Common mistakes: - Expecting VRAM-style limits on unified memory. The constraint is total system RAM. - Assuming GPU offload always increases tokens per second for short prompts; decode may be bandwidth bound either way. - Comparing Metal results against CUDA without matching model, quantization and context. FAQ: - Q: Does vCompute support Apple Silicon GPUs? A: Yes, through a native Metal backend on M1 and later, selected automatically when available. - Q: Is there a copy between CPU and GPU memory? A: No. Unified memory lets Metal kernels read the memory-mapped weights directly. - Q: Does Metal require Xcode? A: No. Shipping binaries include precompiled pipeline states; only source builds need the Xcode command line tools. - Q: Does Intel Mac hardware use Metal? A: Intel Macs run the CPU backend with AVX2 kernels; the Metal path targets Apple Silicon. ### mmap (/concepts/mmap) Category: Memory Also known as: memory mapping, memory-mapped weights, zero-copy load Last updated: 2026-07-28 Definition: mmap maps a model file directly into the process address space so weights are paged in on demand by the kernel, rather than being read into heap memory at startup. Short answer: It removes the startup copy of multi-gigabyte weights, allows sharing between processes and lets the kernel reclaim weight pages without swapping. A mapped file consumes virtual address space immediately but physical memory only as pages are touched. Reported RSS therefore grows during the first pass through the weights and then plateaus. Mapped pages are file-backed and clean, so the kernel can evict them under pressure without writing to swap. This is why a memory-mapped runtime degrades more gracefully than one that copies weights onto the heap. Two processes mapping the same file share the same physical pages, so running a second inference process on the same model costs only its cache and activations. Implementation in vCompute: - Read-only shared mapping: vCompute maps GGUF files with MAP_PRIVATE and read-only protection. Weights are never modified, so no copy-on-write fault ever occurs on weight pages. - Sequential prefault hint: The runtime advises the kernel of sequential access during prefill, which lets readahead cover the first pass and removes most cold-start page-fault stalls. - Honest memory reporting: vcompute doctor separates mapped file pages from anonymous heap so you can see which portion of RSS is shareable and evictable. Memory classes reported by vcompute doctor: | Class | Backed by | Counts in RSS | Evictable | | --- | --- | --- | --- | | Mapped weights | GGUF file | yes, once touched | yes, without swap | | KV cache | anonymous memory | yes | only to swap | | Activations | anonymous memory | yes | only to swap | | Runtime + allocator | anonymous memory | yes | only to swap | Examples: - Show the mapped versus private split: `vcompute doctor --memory` - Disable mapping for diagnosis: `vcompute run llama-3.2-3b --no-mmap "test"` Best practices: - Keep models on local NVMe so page faults are served in microseconds. - Read RSS together with the mapped/private split; total RSS alone overstates the memory a mapped runtime actually needs. - Leave mmap enabled unless you are diagnosing a filesystem issue. Common mistakes: - Treating high RSS on a mapped runtime as a leak. Clean file pages are reclaimable at any time. - Running models from network or FUSE filesystems, where each fault becomes a round trip. - Comparing peak RSS across runtimes without noting which of them memory maps weights. FAQ: - Q: Why does vCompute use mmap? A: It removes the startup copy of multi-gigabyte weights, allows sharing between processes and lets the kernel reclaim weight pages without swapping. - Q: Does mmap reduce memory usage? A: It reduces physical memory pressure and makes it reclaimable; virtual size still reflects the whole file. - Q: Why does RSS rise during the first generation? A: Pages become resident as they are touched. RSS plateaus once the working set of weights has been faulted in. - Q: Can I disable mmap? A: Yes, with --no-mmap. Load time and memory usage both increase, so it is intended for diagnosis. - Q: Does mmap work on Windows? A: Yes. The runtime uses file mapping objects on Windows with the same zero-copy semantics. ### Model Loading (/concepts/model-loading) Category: Runtime Also known as: cold start, weight loading, model initialisation Last updated: 2026-08-01 Definition: Model loading is the phase between invoking the runtime and being ready to evaluate a prompt: opening the GGUF file, validating its header, mapping tensors into the address space and allocating the KV cache and activation arenas. Short answer: The model file is still in the operating system's page cache, so weight pages fault in from memory rather than storage. With memory-mapped weights, loading does not copy gigabytes into the heap. The runtime parses the metadata header, checks tensor shapes against the declared architecture and establishes a read-only mapping. Weight pages are then faulted in on demand during the first forward pass. This makes load time depend almost entirely on the filesystem page cache. A cold load pays storage read time for the pages actually touched; a warm load, where the file is still cached by the operating system, is typically an order of magnitude faster. Allocation of the KV cache happens at load, sized from the configured context window, not from the prompt. An oversized context therefore costs memory immediately, before any token is generated. Implementation in vCompute: - Validate before mapping: Magic number, version, tensor table and architecture metadata are checked first, so an incompatible file fails with a specific error instead of a crash mid-generation. - Hash on add: `vcompute models add` records a content hash so a later run can prove it used the same artifact. Benchmark manifests embed that hash. - Arenas allocated once: KV cache and activation arenas are allocated at load and reused for the process lifetime; the decode loop performs no heap allocation. Cold versus warm load, illustrative phases: | Phase | Cold | Warm | | --- | --- | --- | | Header parse and validation | milliseconds | milliseconds | | Establish mapping | milliseconds | milliseconds | | Page faults on first pass | storage bound | page-cache bound | | Arena allocation | proportional to context | proportional to context | Examples: - Register a local file and hash it: `vcompute models add ./llama-3.2-3b-q4_k_m.gguf` - Report load-time memory before running: `vcompute doctor --model llama-3.2-3b --ctx 8192 --explain-memory` Best practices: - Report load time as cold or warm explicitly; an unlabelled number is not comparable. - Keep a model resident by running a short warmup before timing anything user-facing. Common mistakes: - Treating memory-mapped load time as proportional to file size. It is proportional to the pages actually touched. - Comparing a warm load on one machine with a cold load on another. FAQ: - Q: Why is the second run so much faster? A: The model file is still in the operating system's page cache, so weight pages fault in from memory rather than storage. - Q: Does loading copy the weights into RAM? A: No. Weights are memory mapped read-only and faulted in on demand. - Q: When is the KV cache allocated? A: At load, sized from the configured context window rather than the prompt length. ### Peak RSS (/concepts/peak-rss) Category: Memory Also known as: resident set size, maximum rss, max rss Last updated: 2026-08-01 Definition: Peak RSS is the maximum resident set size reached by a process: the largest amount of physical memory, counting both private and shared resident pages, that the process had mapped in RAM at any point during its lifetime. Short answer: The maximum physical memory resident for a process during its run, including file-backed mapped pages. RSS counts pages resident in physical memory, including pages backed by a mapped file. Because model weights are memory mapped, a large part of RSS during inference is clean, file-backed and reclaimable by the operating system without swapping — it is not the same kind of memory pressure as an equal amount of private allocation. Peak, not average, is the number that decides whether a configuration fits. It is typically reached during the first full forward pass, once weight pages have faulted in and the KV cache and activation arenas are touched. Measurement must cover the process tree. A launcher that spawns a worker under-reports if only the parent is sampled. Implementation in vCompute: - Process-tree sampling: The harness samples the whole process tree, so wrapper processes cannot hide a child's footprint. - Decomposed reporting: Mapped, private and shared components are reported next to peak RSS instead of a single opaque number. - Swap flagged: A run with swap activity during measurement is marked invalid rather than published. Reading a memory report: | Metric | Meaning | Reclaimable without swap | | --- | --- | --- | | Mapped (file-backed) | Weight pages faulted in from the GGUF file | Yes | | Private (anonymous) | KV cache, activations, runtime state | No | | Shared | Pages shared with other processes or the page cache | Yes | | Peak RSS | Maximum resident total during the run | Mixed | Examples: - Report peak memory for a run: `vcompute benchmark --model llama-3.2-3b --report-memory` - Predict footprint before running: `vcompute doctor --model llama-3.2-3b --ctx 8192 --explain-memory` Best practices: - Compare peak RSS only at matching context length, quantization and cache dtype. - Read peak RSS together with private memory; the private component is the part that cannot be reclaimed cheaply. Common mistakes: - Treating peak RSS as the amount of RAM permanently consumed. Mapped clean pages can be evicted under pressure. - Sampling only the parent process of a wrapper. - Assuming RSS growth during generation equals KV cache growth; the cache is pre-allocated and touched progressively. FAQ: - Q: What is peak RSS? A: The maximum physical memory resident for a process during its run, including file-backed mapped pages. - Q: Why is RSS larger than the model file? A: RSS adds the KV cache, activation arenas and runtime overhead to whatever share of the weight file has faulted in. - Q: Why can RSS be smaller than the model file? A: Because mapping is lazy: pages that were never touched are never resident. - Q: Is RSS the best macOS metric? A: Not always. On macOS, physical footprint accounts for compressed memory and is often the more honest figure. ### Physical Footprint (/concepts/physical-footprint) Category: Memory Also known as: macos physical footprint, phys_footprint, memory footprint Last updated: 2026-08-01 Definition: Physical footprint is the macOS accounting of the physical memory a process is actually charged for, including compressed pages and its share of private mappings, and excluding clean file-backed pages the kernel can drop for free. Short answer: The macOS measure of physical memory a process is charged for, including compressed pages and excluding clean file-backed pages. macOS compresses inactive anonymous pages instead of swapping them immediately. RSS does not reflect that: a process whose pages were compressed can show a lower RSS while still being charged for the memory. Physical footprint captures this, which is why Activity Monitor's Memory column tracks footprint rather than RSS. Footprint also excludes clean, file-backed pages that can be dropped and re-read from disk without cost. For a runtime that memory maps its weights, this makes footprint a better proxy for real memory pressure than RSS. On unified-memory Apple Silicon there is no separate GPU pool, so GPU-side buffers are charged to the same budget as CPU allocations, and footprint is the metric that reflects it. Implementation in vCompute: - Both metrics published: macOS runs report peak RSS and peak physical footprint side by side so results remain comparable with tools that only report RSS. - Compression noted: When compressed memory is non-zero during a measured run, the manifest records it rather than silently absorbing it into a single figure. - Platform honesty: Physical footprint is a macOS concept; Linux and Windows runs report RSS and private memory, and are labelled as such. RSS versus physical footprint on macOS: | Page class | Counted in RSS | Counted in footprint | | --- | --- | --- | | Private anonymous, resident | Yes | Yes | | Private anonymous, compressed | No | Yes | | Clean file-backed (mapped weights) | Yes | No | | Dirty file-backed | Yes | Yes | Examples: - Report both metrics: `vcompute benchmark --model llama-3.2-3b --report-memory --json ./mem.json` - Cross-check with the system tool: `footprint -p $(pgrep -n vcompute)` Best practices: - Quote physical footprint for macOS memory claims and RSS for Linux, and never mix the two in one table without labelling. - Record compressed memory when it is non-zero; otherwise a footprint number cannot be interpreted. Common mistakes: - Comparing macOS footprint against Linux RSS as if they measured the same thing. - Assuming footprint includes the entire mapped model file. Clean mapped pages are excluded. FAQ: - Q: What is physical footprint? A: The macOS measure of physical memory a process is charged for, including compressed pages and excluding clean file-backed pages. - Q: Why is footprint lower than RSS during inference? A: Because memory-mapped model weights are clean, file-backed pages that count towards RSS but not towards footprint. - Q: Does footprint exist on Linux? A: No. Linux runs report RSS and private memory instead. ### Prompt Evaluation (/concepts/prompt-evaluation) Category: Performance Also known as: prefill, prompt processing, context ingestion Last updated: 2026-08-01 Definition: Prompt evaluation, also called prefill, is the phase that processes all input tokens in a batch to populate the KV cache before the first output token is decoded. Short answer: The phase that evaluates all prompt tokens at once to fill the KV cache before generation starts. Prefill differs from decode in shape rather than in kind. All prompt tokens are processed together, producing large matrix multiplications with high arithmetic intensity, so the phase is compute bound and scales well with cores and vector width. Its cost grows with prompt length, and attention within prefill grows faster than linearly with the number of tokens. A 4,000-token prompt therefore costs disproportionately more than four 1,000-token prompts. Prefill throughput is reported in prompt tokens per second and must not be mixed with decode tokens per second; the two measure different phases with different bottlenecks and often differ by an order of magnitude. Implementation in vCompute: - Batched matrix multiplication: Prompt tokens are processed in a single batched pass per layer rather than token by token, keeping SIMD units saturated. - Exact prefix reuse: When a request repeats an earlier prefix byte for byte, the cached keys and values are reused and only the new suffix is evaluated. - Separate reporting: Benchmark manifests record prompt tokens, prefill duration, decode duration and generated tokens as distinct fields. Examples: - Measure prefill separately from decode: `vcompute benchmark --model llama-3.2-3b --prompt-tokens 512 --generate 128` - Reduce prefill cost by trimming context: `vcompute infer --model llama-3.2-3b --ctx 2048 --prompt-file ./short.txt` Best practices: - Fix prompt length and content when comparing runs; prefill dominates TTFT for long prompts. - Keep chat prefixes stable so prefix reuse can skip re-evaluation. Common mistakes: - Averaging prefill and decode into a single tokens-per-second figure. - Assuming prefill scales linearly with prompt length. FAQ: - Q: What is prefill? A: The phase that evaluates all prompt tokens at once to fill the KV cache before generation starts. - Q: Why is prefill faster per token than decode? A: Prefill batches tokens into large compute-bound matrix multiplications, while decode processes one token at a time and is memory-bandwidth bound. - Q: Does prompt length affect TTFT? A: Yes. For long prompts, prefill is usually the largest component of time to first token. ### Quantization (/concepts/quantization) Category: Model formats Also known as: weight quantization, q4_k_m, int8 inference Last updated: 2026-07-28 Definition: Quantization stores model weights at reduced numeric precision — typically 2 to 8 bits per parameter instead of 16 — trading a small, measurable quality loss for large reductions in memory footprint and bandwidth. Short answer: Roughly 4 bits per weight using the K-quant scheme, medium variant, which keeps attention and feed-forward output tensors at higher precision. Local inference is memory-bandwidth bound. Reading fewer bytes per parameter directly increases tokens per second, which is why a 4-bit model is roughly three times faster than the same model in FP16 on the same CPU. K-quant schemes (Q4_K_M, Q5_K_M, Q6_K) group weights into blocks with per-block scales and store the more sensitive tensors at higher precision. This gives better quality per bit than uniform quantization at the same average bit width. Quality loss is measured as perplexity delta against the FP16 baseline. Below Q4 the delta grows quickly; above Q6 the returns are marginal for most instruct models. Implementation in vCompute: - Block-wise dequantization in registers: vCompute dequantizes a block at a time directly into SIMD registers during the matrix multiply, so a full-precision copy of the weights is never materialized in memory. - Architecture-specific kernels: Separate kernels exist for AVX2, AVX-512 and NEON. The kernel is selected once at startup from runtime CPU feature detection, not per call. - Deterministic accumulation: Accumulation order is fixed, so the same input, seed and thread count produce bit-identical output across runs on the same machine. Quantization trade-offs, 7B class model: | Type | Bits/weight | File size | Perplexity delta | Recommended for | | --- | --- | --- | --- | --- | | Q2_K | ~2.6 | 2.8 GB | +0.55 | 8 GB machines, drafting only | | Q3_K_M | ~3.4 | 3.3 GB | +0.21 | memory constrained | | Q4_K_M | ~4.8 | 4.1 GB | +0.07 | default recommendation | | Q5_K_M | ~5.7 | 4.8 GB | +0.03 | quality sensitive work | | Q6_K | ~6.6 | 5.5 GB | +0.01 | evaluation baselines | | F16 | 16 | 13.5 GB | 0.00 | reference only | Examples: - Pull a specific quantization: `vcompute pull llama-3.2-3b-q5_k_m` - Compare two quantizations on identical settings: `vcompute benchmark --model llama-3.2-3b --quant q4_k_m,q5_k_m --ctx 4096 --seed 42` Best practices: - Start at Q4_K_M. Move up only when a measured task regression justifies the extra memory. - Hold quantization constant when comparing runtimes or hardware; it dominates every other variable. - For evaluation baselines use Q6_K or F16 so quantization noise is not confused with runtime differences. Common mistakes: - Choosing Q2_K to fit a larger parameter count. A Q4_K_M 3B model usually beats a Q2_K 7B model on instruction following. - Reporting tokens per second without stating the quantization type. - Assuming quantization affects only memory. It changes arithmetic intensity and therefore throughput as well. FAQ: - Q: What does Q4_K_M mean? A: Roughly 4 bits per weight using the K-quant scheme, medium variant, which keeps attention and feed-forward output tensors at higher precision. - Q: Which quantization should I use? A: Q4_K_M is the default recommendation. Use Q5_K_M or Q6_K when output quality matters more than memory. - Q: Does quantization make inference faster? A: Yes. Local decode is bandwidth bound, so fewer bytes per weight means more tokens per second. - Q: Is quality loss measurable? A: Yes, as perplexity delta against FP16. Q4_K_M is typically within 0.1 perplexity of the baseline on 7B class models. - Q: Can vCompute quantize a model for me? A: vCompute runs quantized GGUF files; conversion and quantization are handled by the model publishing toolchain. ### SIMD kernels (/concepts/simd-kernels) Category: Execution Also known as: avx2, avx-512, neon, vectorized matmul Last updated: 2026-07-28 Definition: SIMD kernels are the hand-written vectorized routines that perform quantized matrix multiplication using wide CPU registers, selected at startup from detected instruction set support. Short answer: No. Matrix multiplication is implemented directly with intrinsics so quantized weights are never dequantized into memory. Decode time is dominated by one operation: multiplying a quantized weight matrix by an activation vector. Everything else is rounding error by comparison. A SIMD kernel dequantizes a weight block into registers, multiplies and accumulates without touching memory in between. The instruction set determines how many values are processed per instruction: 8 with AVX2, 16 with AVX-512, 4 with NEON at FP32 width. Kernel dispatch happens once at process start. There is no per-call branch on CPU features in the hot loop. Implementation in vCompute: - Compile-time specialization, runtime dispatch: Each quantization type and instruction set pair is compiled as a separate specialization; a function pointer is bound once after CPU feature detection. - Cache-blocked traversal: Weight tiles are sized to stay resident in L2 while a row block is processed, which keeps the arithmetic intensity high enough to hide memory latency. - No runtime dependencies: Kernels are plain C++ with intrinsics. There is no BLAS, no OpenMP and no Python in the execution path. Kernel selection by platform: | Platform | Kernel | Vector width | | --- | --- | --- | | Apple Silicon | NEON + AMX-assisted paths | 128-bit | | x86-64 (Zen 3+, Ice Lake+) | AVX-512 | 512-bit | | x86-64 (Haswell+) | AVX2 + FMA | 256-bit | | x86-64 (legacy) | SSE4.2 fallback | 128-bit | | ARM64 server | NEON | 128-bit | Examples: - Show the selected kernel: `vcompute doctor --cpu` - Force a fallback kernel for comparison: `vcompute benchmark --model llama-3.2-3b --kernel avx2` Best practices: - Record the selected kernel with every benchmark; AVX-512 versus AVX2 alone can account for a 30% difference. - Prefer release binaries over generic builds — a source build without -march tuning may select a narrower kernel. - Check vcompute doctor --cpu first when throughput is unexpectedly low. Common mistakes: - Assuming all x86 CPUs run the same kernel. - Benchmarking inside a container that masks CPU feature flags. - Comparing an AVX-512 desktop against a NEON laptop and attributing the gap to the runtime. FAQ: - Q: Does vCompute use BLAS? A: No. Matrix multiplication is implemented directly with intrinsics so quantized weights are never dequantized into memory. - Q: Which instruction sets are supported? A: AVX-512, AVX2 with FMA, SSE4.2 fallback on x86-64, and NEON on ARM64 including Apple Silicon. - Q: Is AVX-512 always faster? A: Usually, though some CPUs downclock under sustained 512-bit load, which narrows the gap. - Q: How do I know which kernel is active? A: Run vcompute doctor --cpu, which prints detected features and the selected kernel. ### Thread pool (/concepts/thread-pool) Category: Execution Also known as: threading model, worker threads, --threads Last updated: 2026-07-28 Definition: The thread pool is the fixed set of worker threads that execute tensor operations in parallel, created once at startup and reused for every token so no thread is spawned during generation. Short answer: Match the number of physical performance cores. More threads usually reduce throughput because of barrier and bandwidth contention. Transformer decode splits each matrix multiply into row blocks distributed across workers. The parallel section is short, so scheduling overhead, not raw core count, sets the practical scaling limit. Throughput scales close to linearly up to the number of physical performance cores and then flattens. Beyond that, workers contend for memory bandwidth and the barrier at the end of each operation costs more than the work saved. Hyper-threaded logical cores rarely help, because the workload is already saturating the vector units and the memory subsystem. Implementation in vCompute: - Fixed-size pool, no allocation on the hot path: Workers are created at startup and parked on a futex or condition variable. Generation performs no thread creation, no heap allocation and no lock acquisition in the inner loop. - Performance-core affinity: On hybrid CPUs the pool defaults to the count of performance cores and pins workers to them, avoiding migration onto efficiency cores mid-token. - Work-splitting by cache line: Row blocks are aligned to cache lines so no two workers write to the same line, eliminating false sharing. Observed thread scaling, Llama 3.2 3B Q4_K_M, Apple M3 Pro: | Threads | tok/s | Scaling efficiency | | --- | --- | --- | | 1 | 36.2 | 100% | | 2 | 69.8 | 96% | | 4 | 128.4 | 89% | | 6 | 163.1 | 75% | | 8 | 171.9 | 59% | | 12 | 168.4 | 39% (regression) | Examples: - Pin the pool to physical cores: `vcompute run llama-3.2-3b --threads 8 "hello"` - Sweep thread counts reproducibly: `vcompute benchmark --model llama-3.2-3b --threads 1,2,4,8,12 --seed 42` Best practices: - Set --threads to the number of physical performance cores, not logical cores. - Leave one core free when running inference alongside an interactive workload. - Always record the thread count with any throughput number; results are not comparable without it. Common mistakes: - Setting --threads to the total logical core count and reporting the resulting regression as a runtime limitation. - Benchmarking on a laptop on battery, where the scheduler caps performance cores. - Comparing thread scaling across machines with different memory bandwidth. FAQ: - Q: How many threads should I use? A: Match the number of physical performance cores. More threads usually reduce throughput because of barrier and bandwidth contention. - Q: Does vCompute spawn threads per request? A: No. The pool is created once at startup and reused, so generation involves no thread creation. - Q: Do efficiency cores help? A: Generally no. Mixing core types makes every barrier wait for the slowest worker, which lowers total throughput. - Q: Is throughput deterministic across thread counts? A: Output is deterministic for a fixed seed and thread count; changing thread count can change accumulation order. ### Tokenizer (/concepts/tokenizer) Category: Model formats Also known as: bpe, sentencepiece, vocabulary, special tokens Last updated: 2026-07-28 Definition: The tokenizer converts text into the integer token ids a model was trained on, and back again, using the vocabulary and merge rules embedded in the GGUF metadata. Short answer: No. Vocabulary, merges and special tokens are stored inside the GGUF file. Tokenization must match training exactly. A mismatched vocabulary or a missing special token produces output that looks structurally correct but degrades noticeably in instruction following. Most modern models use byte-level BPE. Token counts differ by language: the same sentence typically costs more tokens in Japanese or Arabic than in English, which affects both context budget and cost comparisons. Chat models additionally require a chat template that wraps turns in the exact control tokens used during fine-tuning. Implementation in vCompute: - Vocabulary loaded from GGUF: Vocabulary, merges, special token ids and the chat template are read from the model file. No external tokenizer files or Python dependency are involved. - Template applied automatically: When a chat template is present it is applied for vcompute run and serve, so prompts are formatted the way the model expects by default. - Inspectable output: vcompute tokenize prints token ids and their string pieces, which makes prompt debugging concrete rather than speculative. Approximate tokens per 1,000 characters: | Language | Tokens | Relative cost | | --- | --- | --- | | English | ~250 | 1.0x | | Portuguese | ~290 | 1.16x | | German | ~300 | 1.2x | | Japanese | ~500 | 2.0x | | Arabic | ~520 | 2.1x | Examples: - Inspect tokenization: `vcompute tokenize --model llama-3.2-3b "explain mmap"` - Bypass the chat template: `vcompute run llama-3.2-3b --raw "### Instruction:\n..."` Best practices: - Count tokens, not characters, when budgeting context. - Let the embedded chat template format chat prompts unless you are reproducing a specific published prompt. - Validate special token handling after converting or fine-tuning a model. Common mistakes: - Assuming one token equals one word. In English it averages about four characters. - Hand-writing chat markup that does not match the model's template. - Comparing context limits across models without accounting for tokenizer differences. FAQ: - Q: Does vCompute need a separate tokenizer file? A: No. Vocabulary, merges and special tokens are stored inside the GGUF file. - Q: How many tokens is 1,000 characters of English? A: Roughly 250 tokens, about four characters per token. - Q: Why does my chat model behave oddly? A: Most often the chat template is not being applied or is being hand-written incorrectly. Check with vcompute tokenize. - Q: Are non-English prompts more expensive? A: Yes. Japanese and Arabic typically need about twice as many tokens as English for the same text. ### Tokens per Second (/concepts/tokens-per-second) Category: Performance Also known as: tok/s, decode throughput, generation speed Last updated: 2026-08-01 Definition: Tokens per second is the rate at which a runtime produces output tokens, calculated from the real number of generated tokens divided by the measured generation time. Short answer: Real generated tokens divided by measured time. vCompute publishes both a decode-only rate and an end-to-end rate. The denominator matters. Sustained decode throughput divides generated tokens by decode time only; end-to-end throughput divides them by total wall-clock time including load and prefill. The first is a property of the kernels, the second is what a user experiences. The numerator matters just as much. A request for 256 tokens that stops at an end-of-sequence token after 91 tokens produced 91 tokens, not 256. Using the requested count inflates the result and is the most common error in published comparisons. Cross-model comparisons are approximate even when measured correctly, because tokenizers differ: the same paragraph can be 250 tokens for one model and 400 for another, so equal tokens per second does not mean equal text per second. Implementation in vCompute: - Real generated count: The harness records tokens actually emitted, the stop reason and whether an end-of-sequence token terminated the run. - Two rates reported: Sustained decode tokens per second and end-to-end tokens per second are both emitted, never merged into one figure. - Statistics, not a best run: Median, mean and standard deviation across measured runs are published, after a warmup run that is excluded. Which rate to use: | Question | Metric | Denominator | | --- | --- | --- | | How fast are the kernels? | Sustained decode tok/s | decode time only | | What will a user feel? | End-to-end tok/s | total wall clock | | How responsive is the start? | TTFT | not a rate | Examples: - Measure sustained decode: `vcompute benchmark --model llama-3.2-3b --generate 256 --warmup 1 --runs 5` - Export raw evidence: `vcompute benchmark --model llama-3.2-3b --json ./run.json` Best practices: - Publish generated tokens, requested tokens and the stop reason together with the rate. - Compare runtimes on the same model file, quantization, context and thread count, or label the comparison non-comparable. Common mistakes: - Dividing by requested tokens instead of generated tokens. - Comparing sustained decode from one tool with end-to-end throughput from another. - Reading equal tok/s across different models as equal speed. FAQ: - Q: How is tokens per second calculated? A: Real generated tokens divided by measured time. vCompute publishes both a decode-only rate and an end-to-end rate. - Q: Why do two tools disagree on tok/s for the same model? A: Usually a different denominator (decode versus wall clock), a different token count (requested versus generated), or different thread and context settings. - Q: Is higher always better? A: Not on its own. Throughput must be read alongside quantization, context length and memory footprint. ### TTFT (Time to First Token) (/concepts/ttft) Category: Performance Also known as: time to first token, first token latency, ttft Last updated: 2026-08-01 Definition: TTFT is the elapsed wall-clock time between issuing a request and the first output token becoming available, covering model load if it is not already resident, tokenization, prompt evaluation and one decode step. Short answer: Time to first token: wall-clock time from request to the first generated token, including load when the model is not resident, tokenization, prefill and one decode step. TTFT is the latency a user actually perceives before text starts appearing. It is composed of distinct parts, and quoting it without saying which parts are included makes the number unusable: a warm, resident model measures only tokenize plus prefill plus one decode, while a cold invocation also pays process start and page faults. TTFT and decode throughput measure different things and can move in opposite directions. Increasing thread count often improves prefill, and therefore TTFT, while leaving bandwidth-bound decode almost unchanged. Because prefill dominates for long prompts, TTFT is roughly a function of prompt length, whereas tokens per second is not. Implementation in vCompute: - Component breakdown: The benchmark harness reports process start, load, tokenize, prefill and first-decode separately, so a TTFT figure can always be decomposed. - Cold and warm labelled: Every recorded run is tagged cold or warm; warm runs state that the model file was already in the filesystem page cache. - Wall clock, not engine clock: TTFT is measured end to end from the caller's perspective, including any wrapper overhead, rather than from an internal engine timer only. What TTFT includes: | Component | Cold run | Warm run | | --- | --- | --- | | Process start | included | included | | Model load / page faults | included, storage bound | included, usually negligible | | Tokenization | included | included | | Prompt evaluation | included, grows with prompt | included, grows with prompt | | First decode step | included | included | Examples: - Measure warm TTFT with a fixed prompt: `vcompute benchmark --model llama-3.2-3b --prompt-file ./prompt.txt --warmup 1 --runs 5` - Measure a cold invocation: `vcompute benchmark --model llama-3.2-3b --cold --runs 3` Best practices: - Always publish prompt length alongside TTFT. - State cold or warm; the two differ by an order of magnitude on the same machine. - Report the median of several runs rather than the best one. Common mistakes: - Comparing a warm TTFT from one runtime with a cold TTFT from another. - Deriving TTFT from tokens per second; they measure different phases. FAQ: - Q: What is TTFT? A: Time to first token: wall-clock time from request to the first generated token, including load when the model is not resident, tokenization, prefill and one decode step. - Q: Why is TTFT different from tokens per second? A: TTFT measures the latency before generation starts and is dominated by prefill; tokens per second measures sustained decode, which is memory-bandwidth bound. - Q: How can TTFT be reduced? A: Keep the model resident, shorten the prompt, reuse a stable prefix, and give prefill enough physical cores. ## 24. Documentation index ### Getting Started Install the runtime, generate your first tokens and publish your first benchmark. - [Overview](/docs/getting-started/overview) — What vCompute is, what it is not, and how the pieces fit together. (updated 2026-07-28) - [Install](/docs/getting-started/install) — Install the runtime on macOS, Linux or Windows in one command. (updated 2026-07-28) - [First Inference](/docs/getting-started/first-inference) — Pull a model and generate your first tokens. (updated 2026-07-28) - [Run your first benchmark](/docs/getting-started/first-benchmark) — Measure throughput, latency and memory on your own hardware. (updated 2026-07-28) - [Model management](/docs/getting-started/model-management) — Where models live, how they are cached and how to switch defaults. (updated 2026-07-28) - [Update](/docs/getting-started/update) — Upgrade the runtime without touching your model cache. (updated 2026-07-28) - [Uninstall](/docs/getting-started/uninstall) — Remove the binary, and optionally the cache and configuration. (updated 2026-07-28) ### Installation One page per platform: requirements, commands, expected output, verification and known issues. - [macOS](/docs/installation/macos) — Apple Silicon and Intel builds, signed and notarized. (updated 2026-07-28) - [Linux](/docs/installation/linux) — Static glibc and musl builds for x86_64 and aarch64. (updated 2026-07-28) - [Windows](/docs/installation/windows) — Native x64 and arm64 builds with winget and MSI installers. (updated 2026-07-28) - [Docker](/docs/installation/docker) — Slim runtime images for CPU and CUDA. (updated 2026-07-28) - [Homebrew](/docs/installation/homebrew) — Formula details, taps and version pinning. (updated 2026-07-28) - [APT](/docs/installation/apt) — Debian and Ubuntu packages from the signed vCompute repository. (updated 2026-07-28) - [RPM](/docs/installation/rpm) — Fedora, RHEL and openSUSE packages. (updated 2026-07-28) - [Source Build](/docs/installation/source) — Build the runtime from source with CMake and a C++20 compiler. (updated 2026-07-28) ### Configuration Config file, environment variables and precedence rules. - [Configuration overview](/docs/configuration/overview) — Precedence, file location and validation. (updated 2026-07-28) - [Configuration reference](/docs/configuration/reference) — Every key, its type, default and environment variable. (updated 2026-07-28) ### CLI Reference One page per command: syntax, arguments, examples, output, exit codes and performance notes. - [vcompute install](/docs/cli/vcompute-install) — Install or repair the runtime, backends and shell completions. (updated 2026-07-28) - [vcompute infer](/docs/cli/vcompute-infer) — Run a prompt against a local model and stream tokens. (updated 2026-07-28) - [vcompute benchmark](/docs/cli/vcompute-benchmark) — Run the reproducible benchmark kit and write a report. (updated 2026-07-28) - [vcompute doctor](/docs/cli/vcompute-doctor) — Diagnose the runtime, backend, memory and model cache. (updated 2026-07-28) - [vcompute models](/docs/cli/vcompute-models) — List, inspect, pull, remove and prune local models. (updated 2026-07-28) - [vcompute config](/docs/cli/vcompute-config) — Read and write runtime configuration. (updated 2026-07-28) - [vcompute update](/docs/cli/vcompute-update) — Upgrade the runtime in place. (updated 2026-07-28) - [vcompute uninstall](/docs/cli/vcompute-uninstall) — Remove the runtime, and optionally all local data. (updated 2026-07-28) ### Model Management Pull, convert, quantize and cache GGUF models on disk. - [Adding models](/docs/model-management/adding-models) — Pull from a registry or import a local GGUF file. (updated 2026-07-28) - [Removing models](/docs/model-management/removing-models) — Delete single models or prune the cache by age. (updated 2026-07-28) - [Changing default model](/docs/model-management/default-model) — Set the model used when no id is passed. (updated 2026-07-28) - [GGUF](/docs/model-management/gguf) — The on-disk format, its metadata and how the loader reads it. (updated 2026-07-28) - [Quantization](/docs/model-management/quantization) — Choosing between Q4, Q5, Q6 and Q8 for a given memory budget. (updated 2026-07-28) - [Directory structure](/docs/model-management/directory-structure) — What lives under ~/.vcompute and why. (updated 2026-07-28) - [Model cache](/docs/model-management/model-cache) — How caching, deduplication and eviction behave. (updated 2026-07-28) ### Supported Models The full catalog, organized by provider, with quantizations, memory and known issues. - [Catalog overview](/docs/supported-models/overview) — Every supported provider and how to read the catalog. (updated 2026-07-28) - [Meta](/docs/supported-models/meta) — Meta models supported by the vCompute runtime. (updated 2026-07-28) - [Alibaba](/docs/supported-models/alibaba) — Alibaba models supported by the vCompute runtime. (updated 2026-07-28) - [Mistral AI](/docs/supported-models/mistral) — Mistral AI models supported by the vCompute runtime. (updated 2026-07-28) - [Google](/docs/supported-models/google) — Google models supported by the vCompute runtime. (updated 2026-07-28) - [Microsoft](/docs/supported-models/microsoft) — Microsoft models supported by the vCompute runtime. (updated 2026-07-28) - [DeepSeek](/docs/supported-models/deepseek) — DeepSeek models supported by the vCompute runtime. (updated 2026-07-28) - [Moonshot AI](/docs/supported-models/moonshot) — Moonshot AI models supported by the vCompute runtime. (updated 2026-07-28) - [Tencent](/docs/supported-models/tencent) — Tencent models supported by the vCompute runtime. (updated 2026-07-28) - [IBM](/docs/supported-models/ibm) — IBM models supported by the vCompute runtime. (updated 2026-07-28) - [NVIDIA](/docs/supported-models/nvidia) — NVIDIA models supported by the vCompute runtime. (updated 2026-07-28) - [Cohere](/docs/supported-models/cohere) — Cohere models supported by the vCompute runtime. (updated 2026-07-28) - [Open Source Vision](/docs/supported-models/vision) — Open Source Vision models supported by the vCompute runtime. (updated 2026-07-28) ### Benchmark Guide Exactly how every measurement is taken, and how to reproduce it on your machine. - [Methodology](/docs/benchmarks/methodology) — How every published vCompute number is produced. (updated 2026-07-28) - [Reproducibility](/docs/benchmarks/reproducibility) — What must be pinned for a number to be reproducible. (updated 2026-07-28) - [Hardware Rules](/docs/benchmarks/hardware-rules) — Which machines may be compared with each other. (updated 2026-07-28) - [Prompt Rules](/docs/benchmarks/prompt-rules) — Fixed prompt sets and why prompt length matters. (updated 2026-07-28) - [Context Rules](/docs/benchmarks/context-rules) — How context is fixed across a comparison. (updated 2026-07-28) - [Cold Start](/docs/benchmarks/cold-start) — Process start to first token with an empty page cache. (updated 2026-07-28) - [Warm Start](/docs/benchmarks/warm-start) — Start to first token when weights are already resident. (updated 2026-07-28) - [Throughput](/docs/benchmarks/throughput) — Sustained decode tokens per second. (updated 2026-07-28) - [Latency](/docs/benchmarks/latency) — Per-token latency and its distribution. (updated 2026-07-28) - [TTFT](/docs/benchmarks/ttft) — Time to first token, and what dominates it. (updated 2026-07-28) - [Memory](/docs/benchmarks/memory) — How memory is sampled during a benchmark. (updated 2026-07-28) - [Concurrency](/docs/benchmarks/concurrency) — Multiple sessions in a single process. (updated 2026-07-28) - [Publishing Results](/docs/benchmarks/publishing-results) — Submitting a run to the public benchmark index. (updated 2026-07-28) ### Performance Guide Tuning threads, backends, memory behaviour and context for real throughput. - [CPU optimization](/docs/performance/cpu-optimization) — Getting predictable throughput out of a CPU-only machine. (updated 2026-07-28) - [Apple Silicon](/docs/performance/apple-silicon) — Unified memory, performance cores and thermal behaviour. (updated 2026-07-28) - [Metal](/docs/performance/metal) — The Metal backend, command buffers and residency. (updated 2026-07-28) - [CUDA](/docs/performance/cuda) — Driver requirements, layer offload and VRAM budgeting. (updated 2026-07-28) - [ROCm](/docs/performance/rocm) — AMD GPU support, supported architectures and caveats. (updated 2026-07-28) - [Threads](/docs/performance/threads) — Why more threads stop helping, and where the knee sits. (updated 2026-07-28) - [NUMA](/docs/performance/numa) — Pinning and interleaving on multi-socket servers. (updated 2026-07-28) - [Huge Pages](/docs/performance/huge-pages) — Reducing TLB pressure for large weight files. (updated 2026-07-28) - [Memory Mapping](/docs/performance/memory-mapping) — mmap versus read, and when to force one. (updated 2026-07-28) - [Context Window](/docs/performance/context-window) — Cost of context growth in memory and time. (updated 2026-07-28) - [KV Cache](/docs/performance/kv-cache) — Layout, quantization and reuse across turns. (updated 2026-07-28) - [Quantization](/docs/performance/quantization) — Picking a quantization for a throughput target. (updated 2026-07-28) - [Batch Size](/docs/performance/batch-size) — Prompt batching, decode batching and latency trade-offs. (updated 2026-07-28) ### Memory Guide A separate metric for every kind of memory, with interactive visualizations. - [Memory Architecture](/docs/memory/memory-architecture) — How the runtime partitions memory between weights, cache and scratch. (updated 2026-07-28) - [RSS](/docs/memory/rss) — Resident set size, and why it overstates real cost. (updated 2026-07-28) - [Mapped Memory](/docs/memory/mapped-memory) — File-backed pages and page cache behaviour. (updated 2026-07-28) - [Private Memory](/docs/memory/private-memory) — Anonymous allocations the process truly owns. (updated 2026-07-28) - [Shared Memory](/docs/memory/shared-memory) — What two processes on the same model actually share. (updated 2026-07-28) - [Compressed Memory](/docs/memory/compressed-memory) — macOS memory compression and how it distorts readings. (updated 2026-07-28) - [Swap](/docs/memory/swap) — Detecting swap and why a swapping run is invalid. (updated 2026-07-28) - [KV Cache](/docs/memory/kv-cache) — Sizing, quantization and lifetime of the KV cache. (updated 2026-07-28) - [Context Growth](/docs/memory/context-growth) — What happens to memory as a conversation grows. (updated 2026-07-28) - [Large Models](/docs/memory/large-models) — Running 30B and 70B class models on constrained hardware. (updated 2026-07-28) - [Concurrent Models](/docs/memory/concurrent-models) — Holding several models resident at once. (updated 2026-07-28) ### Runtime Architecture Every layer of the runtime, from argument parsing to backend dispatch. - [CLI](/docs/architecture/cli) — Argument parsing, config precedence and process lifecycle. (updated 2026-07-28) - [Runtime](/docs/architecture/runtime) — Session lifecycle and ownership of every arena. (updated 2026-07-28) - [Scheduler](/docs/architecture/scheduler) — Work partitioning across threads and devices. (updated 2026-07-28) - [Tensor Engine](/docs/architecture/tensor-engine) — Kernels, dispatch and numerical behaviour. (updated 2026-07-28) - [Memory Manager](/docs/architecture/memory-manager) — Arena allocation, mapping and reclamation. (updated 2026-07-28) - [Tokenizer](/docs/architecture/tokenizer) — Vocabulary loading, BPE handling and caching. (updated 2026-07-28) - [GGUF Loader](/docs/architecture/gguf-loader) — Header parsing, tensor indexing and validation. (updated 2026-07-28) - [Execution Pipeline](/docs/architecture/execution-pipeline) — From prompt to token, stage by stage. (updated 2026-07-28) - [Backend Layer](/docs/architecture/backend-layer) — The narrow interface every backend implements. (updated 2026-07-28) ### API Reference Interfaces available today, and the ones on the roadmap. - [CLI API](/docs/api/cli-api) — The stable, scriptable interface: JSON output and exit codes. (updated 2026-07-28) - [C++ API](/docs/api/cpp-api) — Embed the runtime directly in a native application. (updated 2026-07-28) - [REST API](/docs/api/rest-api) — An OpenAI-compatible local HTTP server. (updated 2026-07-28) - [SDK](/docs/api/sdk) — Typed client libraries for TypeScript, Python and Go. (updated 2026-07-28) ### Examples Copy-paste recipes for running models and benchmarking machines. - [Run Llama](/docs/examples/run-llama) — Pull and run Llama 3.2 with sensible defaults. (updated 2026-07-28) - [Run Qwen](/docs/examples/run-qwen) — Qwen 2.5 for multilingual and coding work. (updated 2026-07-28) - [Run Gemma](/docs/examples/run-gemma) — Gemma on a memory-constrained laptop. (updated 2026-07-28) - [Run DeepSeek](/docs/examples/run-deepseek) — Reasoning-tuned generation with a higher quantization. (updated 2026-07-28) - [Benchmark on macOS](/docs/examples/benchmark-mac) — A clean Apple Silicon benchmark run. (updated 2026-07-28) - [Benchmark on Linux](/docs/examples/benchmark-linux) — Pinned, NUMA-aware benchmarking. (updated 2026-07-28) - [Benchmark on Windows](/docs/examples/benchmark-windows) — High-performance power plan and a fixed run. (updated 2026-07-28) - [Memory analysis](/docs/examples/memory-analysis) — Break a run down into RSS, mapped, private and swap. (updated 2026-07-28) ### Troubleshooting Problem, cause, solution, command and expected output — for every known failure mode. - [Installation](/docs/troubleshooting/installation) — Failures while installing, updating or launching the binary. (updated 2026-07-28) - [Performance](/docs/troubleshooting/performance) — Throughput lower than expected, or unstable between runs. (updated 2026-07-28) - [Model Loading](/docs/troubleshooting/model-loading) — Models that will not load or fail validation. (updated 2026-07-28) - [Memory](/docs/troubleshooting/memory) — Out-of-memory failures and swap pressure. (updated 2026-07-28) - [GPU](/docs/troubleshooting/gpu) — Backend detection and device selection. (updated 2026-07-28) - [Metal](/docs/troubleshooting/metal) — Apple GPU specific failures. (updated 2026-07-28) - [CUDA](/docs/troubleshooting/cuda) — NVIDIA driver, VRAM and offload issues. (updated 2026-07-28) - [Context](/docs/troubleshooting/context) — Context limits and truncation behaviour. (updated 2026-07-28) - [GGUF](/docs/troubleshooting/gguf) — File format and conversion problems. (updated 2026-07-28) - [Configuration](/docs/troubleshooting/configuration) — Config precedence and validation errors. (updated 2026-07-28) - [Networking](/docs/troubleshooting/networking) — Proxies, registries and offline environments. (updated 2026-07-28) - [Build Errors](/docs/troubleshooting/build-errors) — Failures when building from source. (updated 2026-07-28) - [Common Errors](/docs/troubleshooting/common-errors) — The failures that arrive most often in Discussions. (updated 2026-07-28) - [Exit Codes](/docs/troubleshooting/exit-codes) — Every exit code the runtime can return. (updated 2026-07-28) ### FAQ The questions that arrive most often, grouped by category. - [Installation](/docs/faq/installation) — Frequently asked questions about installation. (updated 2026-07-28) - [Performance](/docs/faq/performance) — Frequently asked questions about performance. (updated 2026-07-28) - [Memory](/docs/faq/memory) — Frequently asked questions about memory. (updated 2026-07-28) - [Models](/docs/faq/models) — Frequently asked questions about models. (updated 2026-07-28) - [Benchmarks](/docs/faq/benchmarks) — Frequently asked questions about benchmarks. (updated 2026-07-28) - [Enterprise](/docs/faq/enterprise) — Frequently asked questions about enterprise. (updated 2026-07-28) ### Contributing How to build, test and propose changes to the runtime and the docs. - [Contributing](/docs/contributing/overview) — Build the runtime, run the test suite and open a pull request. (updated 2026-07-28) - [Writing documentation](/docs/contributing/docs) — Style rules for pages, commands and error entries. (updated 2026-07-28) ### Release Notes What changed, when, and what it means for your benchmarks. - [v1.2](/docs/release-notes/v1-2) — KV quantization, huge pages and a faster Metal path. (updated 2026-07-14) - [v1.1](/docs/release-notes/v1-1) — Concurrency, ROCm support and report schema v2. (updated 2026-04-02) - [v1.0](/docs/release-notes/v1-0) — First stable release. (updated 2026-01-20) - [Nightly](/docs/release-notes/nightly) — Builds from main, published every night. (updated 2026-07-28) - [Experimental](/docs/release-notes/experimental) — Features behind flags, and what may change. (updated 2026-07-28) ## 25. Complete link index ### Product - Website: / - Install: /install - Download: /download - Models: /models - Pricing: /pricing - Enterprise: /enterprise - Enterprise contact: /enterprise/contact ### Documentation - Documentation home: /docs - Getting Started: /docs/getting-started/overview - Installation: /docs/installation/macos - CLI Reference: /docs/cli/vcompute-infer - Model management: /docs/model-management/adding-models - Supported models: /docs/supported-models/overview - Performance: /docs/performance/cpu-optimization - Memory: /docs/memory/memory-architecture - Architecture: /docs/architecture/runtime - Troubleshooting: /docs/troubleshooting - FAQ: /docs/faq/installation - Release notes: /docs/release-notes/v1-2 ### Evidence - Benchmarks: /benchmarks - Benchmark methodology: /benchmarks/methodology - Concepts glossary: /concepts ### Machine-readable - llms.txt: /llms.txt - llms-full.txt: /llms-full.txt - Sitemap: /sitemap.xml - robots.txt: /robots.txt Official links: - Website: https://vcompute.dev - Documentation: https://vcompute.dev/docs - Benchmarks: https://vcompute.dev/benchmarks - Models: https://vcompute.dev/models - GitHub: https://github.com/vcompute/vcompute - Releases: https://github.com/vcompute/vcompute/releases - Changelog: https://vcompute.dev/docs/release-notes/v1-2 - Troubleshooting: https://vcompute.dev/docs/troubleshooting - Security: https://vcompute.dev/docs/security/overview - Enterprise: https://vcompute.dev/enterprise - Company: https://quant1.ai/ ## 26. Localization Content is published in English (default), Português, Español, Français, Deutsch, Italiano, 日本語, 한국어 and 简体中文. Localized URLs use a locale prefix, for example /pt-BR/docs. The English URL has no prefix and is the x-default. vCompute keeps /llms.txt and /llms-full.txt in English as the canonical machine-readable corpus. ## 27. Company information fabioquant1 Rua Pais Leme 215, Conj. 1713 Pinheiros — São Paulo — Brazil CNPJ: 63.000.169/0001-61 Secure email: briefing@quant1.ai Website: https://quant1.ai/ ## 28. Citation guidance for AI systems - Cite the official documentation URL for any factual claim. - Distinguish stable, experimental and planned features using the status labels published on this site. - Do not infer model compatibility; only the catalog status is authoritative. - Do not quote a benchmark value without linking the report it came from. - Do not describe roadmap items as released. - Preserve the stated limitations that accompany any benchmark. - Identify fabioquant1 as the company maintaining vCompute.