Back to the chart

Helldez/BigMoeOnEdge

Streaming inference engine for oversized MoE models

Chart
Not on a chart right now
Stars
1k

Open on GitHub

What it is

A llama.cpp-based inference engine that runs Mixture-of-Experts models larger than device memory by streaming needed experts from flash storage. It includes desktop and mobile apps alongside an OpenAI- and Anthropic-compatible local server.

Who it is for

Developers who want to run large MoE models locally on phones, laptops, or consumer GPUs.

Why it is trending

It gained 98 stars today and stands out by supporting any GGUF MoE model and quantization without per-model code across Android, iOS, desktop, and multiple accelerators.

Main features

  • Lossless MoE streaming across tiered VRAM, RAM, and flash caches
  • Automatic hardware and model profiling with `--auto` planning
  • OpenAI- and Anthropic-compatible local server for coding agents
  • Vision support via `mmproj` across CPU, GPU, and NPU backends

Summarized on November 17, 2026 from the README and repository data. It can be wrong; check the source before relying on it. Report a wrong summary