SkewAdam cuts MoE optimizer memory to 1.29 GB, beating AdamW on perplexity

A technical analysis of SkewAdam, a tiered optimizer that drops peak MoE training memory to 31.3 GB, enabling 7B-class mixture-of-experts training on single 40 GB GPUs.

The answer up front

SkewAdam is a specialized optimizer designed for training Mixture-of-Experts (MoE) language models. By allocating different optimizer states to the dense backbone, experts, and router, it slashes optimizer memory from 50.6 GB to 1.29 GB on a 6.78B-parameter model. For indie AI founders and small engineering teams, this is a clear pick if you are pre-training or fine-tuning MoE models on budget hardware like single 40 GB GPUs. Skip it if you are training standard dense models, as its architectural assumptions rely entirely on the highly skewed parameter distribution of MoEs. The bottom line is that SkewAdam makes MoE training accessible on commodity hardware without sacrificing convergence.

Methodology

This review evaluates SkewAdam based on the research paper "Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training" published on July 22, 2026. Because this is a newly published paper, our analysis draws directly on the authors' published claims and empirical benchmarks; independent replication of these training runs is pending. We cover the technical mechanism of the tiered state allocation, the memory footprint reductions, and the comparative validation perplexity benchmarks against AdamW, Muon, Lion, and Adafactor on an 82-million-token run. We do not cover long-term training stability beyond the reported 82M tokens, edge-case behavior on non-MoE architectures, or performance on hardware architectures other than the unspecified 40 GB accelerators referenced in the paper.

What it does

Tiered state allocation

SkewAdam splits the parameter populations of an MoE model into three distinct tiers, applying different optimizer states to each. For the dense backbone, which represents 5% of the parameters, it maintains float32 momentum and a factored second moment. For the experts, making up 95% of the parameters, it drops momentum entirely and uses only a factored second moment. For the router, representing less than 0.01% of the parameters, it maintains an exact second moment. This targeted allocation addresses the massive memory waste of applying uniform optimizer states across highly asymmetric parameter pools.

Drastic memory reduction

In the paper's benchmark on a 6.78B-parameter MoE model with 12.6 GB of bfloat16 weights, standard AdamW requires 50.6 GB of memory just for its first and second moments. SkewAdam reduces this optimizer state to 1.29 GB, which is 2.6% of AdamW's footprint. This drop reduces peak training memory from 81.4 GB to 31.3 GB. This shift is critical because it brings the peak memory requirement comfortably within the 40 GB limit of widely available, lower-cost accelerators.

Router load balancing

The optimizer stabilizes the router by keeping an exact second moment for its tiny parameter set. According to the authors, this specific allocation allows the router load balance to settle to within 1% of its uniform floor. This addresses a common failure mode in MoE training where routers fail to distribute tokens evenly across experts, leading to underutilized parameters and poor training efficiency.

What's interesting and what's not

The editorial assessment

What is highly compelling about SkewAdam is that the memory savings do not come at the cost of accuracy. In fact, the paper reports that SkewAdam achieved a validation perplexity of 108.4 over 82M tokens, outperforming AdamW (126.8), Muon (120.2), and Lion (393.7). Even when the baselines' learning rates were swept and tuned, the best AdamW run only reached 118.5. The authors' ablation study reveals an important detail: the tiered allocation itself is not what improves perplexity. A tier ablation using twenty times the state achieved the same perplexity. Instead, the accuracy gains come from retaining momentum where it matters, while the tiered structure simply buys memory efficiency at zero accuracy cost. This is a refreshing departure from optimizers like Adafactor, which drops momentum globally to save memory but plateaus 40 points behind SkewAdam.

What is missing

What is missing from the paper is a long-term training run. While 82M tokens is sufficient to show early convergence trends, it is a tiny fraction of a modern pre-training regime, which typically spans trillions of tokens. We do not know if the factored second moments on the experts will introduce gradient accumulation issues or scaling bottlenecks later in training. Additionally, the paper does not specify the exact hardware used for the 40 GB accelerator benchmarks, leaving open questions about real-world wall-clock time and potential overhead from managing three distinct optimizer states in PyTorch.

Pricing

SkewAdam is an open-source academic research project. As of July 2026, there are no licensing fees or commercial tiers. The code is typically made available via public GitHub repositories associated with the paper's release.

Verdict

SkewAdam is a highly promising optimizer for teams training MoE models on limited hardware budgets. If you are actively training or fine-tuning MoEs and are bottlenecked by the 80 GB VRAM requirement of standard AdamW, SkewAdam is a clear pick. It successfully compresses optimizer state memory by over 97% while maintaining, and even improving, convergence speed. However, if you are training standard dense models, skip it. Its entire design relies on the 95-to-5 ratio of expert-to-backbone parameters. For MoE practitioners, it represents a major step forward in democratizing large-scale training.

What we'd test next

In a v2 evaluation, we would benchmark SkewAdam on a cluster of NVIDIA A100 (40GB) or L40S GPUs to measure actual step latency. Managing three separate optimizer states could introduce kernel launch overhead that offsets the memory savings. We would also run a longer training run of at least 10B tokens to verify that the factored second moments do not cause divergence or gradient explosions in later training phases.

The investor read

For investors, SkewAdam signals a critical shift in the AI infrastructure layer: the optimization of MoE architectures is moving from brute-force compute scaling to highly specialized, hardware-aware algorithmic design. As compute costs remain the primary bottleneck for AI startups, tools that dramatically lower the hardware floor (such as enabling 7B-class MoE training on single 40 GB GPUs instead of requiring 80 GB clusters) directly expand the addressable market of builders. While SkewAdam itself is an open-source research artifact, the underlying thesis of "tiered state allocation" is highly investable. We expect to see commercial training frameworks and managed orchestration platforms integrate these tiered optimization techniques to offer lower-cost fine-tuning tiers. Companies building proprietary, hardware-optimized training compilers that can automate these state allocations across arbitrary model architectures are the key venture-scale plays to watch in this space.

Pull quote: “The bottom line is that SkewAdam makes MoE training accessible on commodity hardware without sacrificing convergence.”