Krasis 1.0 ships Rust engine; streams large LLMs on consumer GPUs

A technical analysis of Krasis 1.0, an LLM runtime designed to stream models from system RAM to VRAM, bypassing Python GIL bottlenecks to run 35B+ models on consumer hardware.

Krasis 1.0 is built for developers and local LLM enthusiasts who want to run large models (35B to 122B parameters) on consumer-grade GPUs with limited VRAM. If you are trying to run Qwen3.6-35B on an 8GB RTX 3070 Mobile laptop or an 80B model on a 20GB RTX A4500, Krasis offers a viable path. Skip this runtime if you have dedicated multi-GPU clusters or require enterprise-grade API latencies. The bottom line is that Krasis achieves usable local speeds by streaming weights through VRAM and dropping Python from the execution path.

Methodology

This v0 review draws on the developer's published claims at https://www.reddit.com/r/LocalLLaMA/comments/1tpyqng/krasis_update_qwen3635ba3b_q4_at_reading_speed_1x/ (observed May 2026). Independent benchmarks are pending. This review covers the architectural shift to 100% Rust in the hot path, sensitivity-aware HQQ attention, memory footprint reductions, and the new 4-bit and 6-bit KV cache implementations. It does not cover independent latency validation, long-term stability, or edge-case accuracy drift on complex reasoning tasks.

Streaming weights through VRAM

Krasis is an LLM runtime designed specifically for running models that do not fit entirely into VRAM. Instead of relying on slow CPU offloading, Krasis streams the model through VRAM from system RAM. The engine treats prefill and decode as separate architectures, applying optimized use cases to each phase. In the v1.0 release, the developer reports that the runtime no longer requires twice the quantized model size in system RAM, reducing the requirement to 1x plus some operational overhead.

Rust-native execution engine

A major architectural change in v1.0 is the transition to all-Rust execution. The developer, mrstoatey, removed Python entirely from the hot path. The developer found that the Python Global Interpreter Lock (GIL) was causing frequent slowdowns and processing difficulties. While Python is still used for initial pre-processing, the model execution phase is now 100% Rust, which the developer claims results in faster execution speeds.

Sensitivity-aware HQQ attention

Krasis has dropped AWQ attention in favor of Sensitivity-Aware HQQ (Half-Quadratic Quantization) attention at 4, 6, or 8 bits. AWQ required running models in BF16 to generate a template, which was difficult for users without high-end hardware. HQQ assets are built by mathematically assessing the model directly, eliminating the need for pre-built templates. During this assessment, Krasis estimates which areas of the model are most sensitive to quantization and can mix precision (for example, allocating 90% to HQQ4 and 10% to HQQ6) to maintain accuracy while keeping memory usage low.

Optimized KV caching

The runtime introduces a new 4-bit and 6-bit KV cache implementation. This replaces Polar4, which was based on TurboQuant and dropped due to insufficient accuracy. The new KV cache does not require FP8 instructions, making it fully compatible with Ampere cards like the RTX 3000 series. The developer claims to have thoroughly tested this cache for accuracy against BF16 baselines, measuring exact match length of output, top-k containment, perplexity, and distribution drift.

What is interesting

The elimination of the Python GIL in the hot path is a highly practical engineering decision. Many local runtimes suffer from orchestration overhead, and moving to pure Rust addresses this bottleneck directly. Dropping AWQ for sensitivity-aware HQQ is another major usability win, as it democratizes the quantization process for users who cannot generate BF16 templates. Furthermore, bringing 4-bit and 6-bit KV caching to Ampere cards without requiring FP8 instructions extends the lifespan of older consumer hardware.

What is not

The published benchmark numbers represent the best throughput obtained during testing, not the average across all prompt lengths. The developer notes that prefill throughput scales up with larger inputs, while decode throughput reduces with larger outputs. For example, the reported 12.48 tokens per second decode speed for Qwen3.6-35B-A3B on an 8GB RTX 3070 Mobile laptop is a peak figure. Real-world average performance under sustained, long-context use cases remains unverified and is likely lower.

Pricing

Krasis is an open-source project available for free. This pricing snapshot is accurate as of May 2026.

Verdict

Krasis 1.0 is a compelling choice for developers running large local LLMs on constrained hardware. If you are limited to a single consumer GPU, Krasis's Rust-native streaming architecture and sensitivity-aware HQQ attention make it a superior alternative to standard llama.cpp offloading. It is a highly specialized tool that prioritizes memory efficiency over raw, unconstrained speed.

What we would test next

In our next evaluation, we would run a standardized benchmark suite (such as GSM8K) to verify if the sensitivity-aware mixed precision actually preserves reasoning capabilities compared to uniform quantization. We also plan to measure average throughput across sustained multi-turn conversations to see how the streaming architecture handles KV cache growth under long-context pressure.

The investor read

Krasis highlights a growing trend in the local AI ecosystem: optimizing consumer-grade hardware to run enterprise-grade models. As frontier models grow larger, the bottleneck is no longer compute, but memory bandwidth and VRAM capacity. Runtimes like Krasis that bypass the Python GIL and use hybrid quantization (like sensitivity-aware HQQ) represent a threat to cloud-only API providers. If developers can run 35B to 80B models locally at "reading speed" on a standard laptop, the marginal cost of inference drops to zero for many agentic workflows. While Krasis itself is a bootstrapped, open-source project, the underlying techniques (Rust-native execution, dynamic VRAM streaming, and mixed-precision KV caching) are highly investable areas. We expect to see these optimizations consolidated into larger commercial local-first platforms.

Pull quote: “A major architectural change in v1.0 is the transition to all-Rust execution.”