Benchmarking LLMs for RISC-V Edge: Identifying 100M-200M Parameter INT8 Models

This review identifies suitable 100M-200M parameter, INT8 quantized LLM/SLM models from Hugging Face for benchmarking custom RISC-V controllers with vector/matrix units.

TL;DR Best for: Hardware engineers and researchers building custom RISC-V controllers with specialized vector/matrix units, seeking specific small, quantized LLM/SLM models for performance benchmarking on resource-constrained edge devices. Skip if: You require off-the-shelf, high-performance LLM inference solutions for cloud or desktop environments, or if your focus is on models larger than 200M parameters or different quantization schemes. Bottom line: For RISC-V edge device benchmarking, bert-base-uncased, EleutherAI/pythia-160m, and facebook/opt-125m offer robust starting points, providing well-documented architectures within the target parameter range for INT8 quantization.

METHODOLOGY This v0 review draws on a founder's specific request for benchmarking models, as published on Reddit by user neuroticnetworks1250 on 2026-05-27. Independent benchmarks are pending. Update cadence: re-tested when claims diverge from observed behavior or when new, more suitable models emerge. The models identified—bert-base-uncased (110M parameters), EleutherAI/pythia-160m (160M parameters), and facebook/opt-125m (125M parameters)—are representative examples found on Hugging Face that fit the specified 100M-200M parameter range for LLM/SLM. This review covers the identification of these models, their associated research papers, and their general suitability for INT8 quantization on resource-constrained edge devices. What's not covered includes actual performance benchmarks on the founder's specific RISC-V hardware, long-term workflow integration, or a comprehensive comparison of all possible small LLM architectures beyond the parameter and quantization criteria.

WHAT IT DOES

Small parameter count for edge devices

The core requirement is for models between 100M and 200M parameters. This range is critical for deploying language models on resource-constrained edge devices, where memory footprint and computational complexity are primary bottlenecks. Models like bert-base-uncased (110M), facebook/opt-125m (125M), and EleutherAI/pythia-160m (160M) fit this criterion. These models represent different architectural lineages (encoder-only for BERT, decoder-only for OPT and Pythia), offering varied computational graphs for hardware testing. Their relatively small size allows for easier fitting into limited on-chip memory and faster inference cycles, making them ideal candidates for initial hardware validation and optimization.

INT8 quantization for efficiency

INT8 quantization is a crucial technique for reducing model size and accelerating inference on hardware that supports 8-bit integer operations. The founder's explicit mention of INT8 indicates a focus on maximizing computational efficiency and minimizing memory bandwidth requirements. While these base models are typically released in FP32 or FP16, they are well-understood architectures with established methods for post-training quantization (PTQ) or quantization-aware training (QAT) to INT8. The availability of these models on Hugging Face ensures that researchers can access pre-trained weights and apply standard quantization toolchains (e.g., Hugging Face Optimum, ONNX Runtime, TVM) to generate INT8 versions suitable for their custom RISC-V hardware.

Hugging Face availability and associated papers

All recommended models are readily available on the Hugging Face Hub, providing easy access to pre-trained weights, tokenizers, and configuration files. This platform also often links directly to the associated research papers, which are essential for understanding the model's architecture, training methodology, and reported performance characteristics. For bert-base-uncased, the foundational paper is "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Devlin et al. (2018). For facebook/opt-125m, refer to "OPT: Open Pre-trained Transformer Language Models" by Zhang et al. (2022). For EleutherAI/pythia-160m, the relevant work is "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling" by Biderman et al. (2023). These resources provide the necessary technical depth for hardware-software co-design efforts.

WHAT'S INTERESTING / WHAT'S NOT

What's interesting about this request is the founder's specific focus on a custom RISC-V controller with vector/matrix units, and the consideration of a dedicated Softmax unit if RVV instructions become a bottleneck. This indicates a sophisticated, ground-up approach to hardware-software co-design for AI acceleration at the edge. The explicit parameter range (100M-200M) and INT8 quantization target are pragmatic choices for achieving meaningful performance on highly constrained devices. It highlights a growing trend where hardware architects are not just optimizing for general-purpose compute but are tailoring silicon directly to the needs of specific AI model classes and inference patterns. The choice of LLM/SLM models, rather than vision models, further points to the increasing demand for on-device natural language processing capabilities.

What's not explicitly covered in the founder's request, but is critical for this type of benchmarking, is the specific dataset or task used for evaluation. While the models are identified, their performance characteristics are highly dependent on the downstream task (e.g., text classification, question answering, summarization). Without a defined benchmark task, raw inference speed on a generic input may not fully reflect real-world utility. Additionally, the founder's query doesn't delve into the specific quantization scheme (e.g., static vs. dynamic, symmetric vs. asymmetric) or the calibration dataset used, which can significantly impact INT8 model accuracy and hardware compatibility. The challenge of reproducible benchmarking on custom, early-stage hardware also remains a significant hurdle, as toolchains and drivers may still be in flux.

PRICING The identified models (bert-base-uncased, EleutherAI/pythia-160m, facebook/opt-125m) are open-source and freely available for download and use from the Hugging Face Hub. There are no direct costs associated with acquiring or using these model weights. Costs would arise from compute resources for fine-tuning, quantization, or running inference on custom hardware. Pricing snapshot date: 2026-05-27.

VERDICT For a founder building a RISC-V controller with vector/matrix units and targeting 100M-200M parameter INT8 quantized LLM/SLM models, bert-base-uncased, EleutherAI/pythia-160m, and facebook/opt-125m are the best starting points. These models offer diverse architectures within the specified parameter range, are well-documented with associated research papers, and are readily available on Hugging Face. Their established presence in the research community means robust support for quantization techniques, making them highly suitable for hardware-software co-design and performance benchmarking on resource-constrained edge devices. The choice between them depends on whether the founder prioritizes encoder-only (BERT) or decoder-only (Pythia, OPT) architectures for their specific application and hardware optimization goals.

WHAT WE'D TEST NEXT Our next steps would involve establishing a reproducible benchmarking environment for these models on a simulated or actual RISC-V platform. We would quantify the performance impact of INT8 quantization on each model, measuring latency, throughput, and energy consumption across various batch sizes. A critical test would be to evaluate the founder's hypothesis regarding a dedicated Softmax unit, benchmarking its performance against RVV instructions. We would also explore the trade-offs between different quantization techniques (e.g., static vs. dynamic) and their impact on model accuracy for a specific downstream task. Finally, we would investigate the feasibility of integrating these models with common edge AI frameworks like TVM or ONNX Runtime on RISC-V.

Pull quote: “For a founder building a RISC-V controller with vector/matrix units and targeting 100M-200M parameter INT8 quantized LLM/SLM models, bert-base-uncased, EleutherAI/pythia-160m, and facebook/opt-125m are the best starting points.”