Local Inference

GPT4All

Nomic AI

Desktop application from Nomic for running open LLMs locally on consumer hardware, with private chat and LocalDocs for asking questions about your own files without data leaving your machine.

• Open source
• Local-first
• Privacy-first

Jan

Menlo Research

Recommended

Open-source ChatGPT alternative that runs 100 percent offline on your computer. Desktop app for downloading and chatting with local models, with optional connections to cloud providers.

• Open source
• Local-first
• Privacy-first
• Self-hostable

llama.cpp

ggml-org (Georgi Gerganov)

Recommended

LLM inference engine in C/C++ with minimal dependencies. Runs quantized GGUF models on CPU and GPU across Apple Silicon, x86 and CUDA, and powers many local AI applications.

• Open source
• Local-first
• Privacy-first
• Self-hostable

LM Studio

LM Studio

Desktop app for discovering, downloading and running local LLMs such as Llama, Qwen and Gemma. Includes a chat interface and an OpenAI-compatible local server. Free to use but not open source.

• Local-first
• Privacy-first

Ollama

Ollama

Recommended

Run open large language models locally with a simple CLI and REST API. One-line installs for Llama, Qwen, Gemma, DeepSeek and hundreds of other models on macOS, Linux and Windows.

• Open source
• Local-first
• Privacy-first
• Self-hostable

vLLM

vLLM Project

Recommended

High-throughput, memory-efficient inference and serving engine for LLMs. Uses PagedAttention and continuous batching to serve open models at scale behind an OpenAI-compatible API.

• Open source
• Local-first
• Privacy-first
• Self-hostable

whisper.cpp

ggml-org (Georgi Gerganov)

Recommended

Port of OpenAI's Whisper speech recognition model in plain C/C++. Fast offline transcription on CPU and Apple Silicon with no Python dependency; widely embedded in local dictation and transcription apps.

• Open source
• Local-first
• Privacy-first
• Self-hostable