Hub documentation

Local Agents with llama.cpp

Hub

API docs

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Local Agents with llama.cpp

You can run a coding agent entirely on your own hardware. Several open-source agents can connect to a local llama.cpp server to give you an experience similar to Claude Code or Codex — but everything runs on your machine.

Getting Started

1. Set Your Local Hardware

Set your local hardware so it can show you which models are compatible with your setup.

Go to huggingface.co/settings/hardware and configure your local hardware profile. Then select llama.cpp in your Local Apps settings, as this will be the engine you’ll use.

2. Find a Compatible Model

Browse for Llama.cpp-compatible models.

3. Launch the llama.cpp Server

On the model page, click the “Use this model” button and select llama.cpp. It will show you the exact commands for your setup. The first step is to start a llama.cpp server, e.g.

llama-server -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M

This downloads the model and starts an OpenAI-compatible API server on your machine. See the llama.cpp guide for installation instructions.

4. Connect Your Agent

Pick one of the agents below and follow the setup instructions.

Pi

Pi is the agent behind OpenClaw and is now integrated directly into Hugging Face, giving you access to thousands of compatible models.

Install Pi:

npm install -g @mariozechner/pi-coding-agent

Then add your local model to Pi’s configuration file at ~/.pi/agent/models.json:

{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ggml-org-gemma-4-26b-4b-gguf"
        }
      ]
    }
  }
}

Start Pi in your project directory:

pi

Pi connects to your local llama.cpp server and gives you an interactive agent session.

Demo

Enabling vision support

For vision-capable models, add "input": ["text", "image"] to the model entry in ~/.pi/agent/models.json:

"models": [
  {
    "id": "unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_XL",
    "input": ["text", "image"]
  }
]

Browse vision-language models compatible with Pi.

OpenClaw

OpenClaw works locally with llama.cpp. You can set your model via the onboard command:

openclaw onboard --non-interactive \
  --auth-choice custom-api-key \
  --custom-base-url "http://127.0.0.1:8080/v1" \
  --custom-model-id "ggml-org-gemma-4-26b-a4b-gguf" \
  --custom-api-key "llama.cpp" \
  --secret-input-mode plaintext \
  --custom-compatibility openai \
  --accept-risk

You can also run openclaw onboard interactively, select custom-compatibility with openai, and pass the same configuration.

Local Memory Search for OpenClaw

You can run local embedding models with Llama.cpp for your agent’s memory search. To do so, make sure to have node-llama-cpp.

npm i node-llama-cpp

Here’s an example snippet to run quantized EmbeddingGemma-300M locally for memory search. OpenClaw automatically downloads and serves the model with the command below.

openclaw config set agents.defaults.memorySearch.provider local
openclaw config set agents.defaults.memorySearch.local.modelPath "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf"

Restart the gateway and validate.

openclaw gateway restart
openclaw memory status
# Memory Search (main)
# Provider: local (requested: local)
# Model: hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf

Hermes Agent

Hermes Agent works locally with llama.cpp. Define a default config as:

model:
  provider: custom
  default: ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M
  base_url: http://127.0.0.1:8080/v1
  api_key: llama.cpp

custom_providers:
  - name: Local (127.0.0.1:8080)
    base_url: http://127.0.0.1:8080/v1
    api_key: llama.cpp
    model: ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M

### Local Memory Search for Hermes Agent

Hermes Agent consumes semantic search models through endpoints. Once you get your preferred embedding model up on endpoint 8080 with llama.cpp or the inference engine of your choice, add the following to ~/.hermes/config.yaml.

auxiliary:
  session_search:
    base_url: "http://127.0.0.1:8080/v1"
    api_key: "no-key-required"
    model: "local-llama" # your model alias
    timeout: 90
    max_concurrency: 1

Check if this works, none - built-in only shows that no other memory plug-ins are used. The output below shows that local serving is active.

$ hermes memory status
# Memory status
#────────────────────────────────────────
#  Built-in:  always active
#  Provider:  (none — built-in only)

OpenCode

OpenCode works locally with llama.cpp. Define a ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "gemma-4-26b-4b-it": {
          "name": "Gemma 4 (local)",
          "limit": {
            "context": 128000,
            "output": 8192
          }
        }
      }
    }
  }
}

How It Works

The setup has two components running locally:

llama.cpp server — Serves the model as an OpenAI-compatible API on localhost.
Your agent — The agent process that sends prompts to the local server, reasons about tasks, and executes actions.

┌─────────┐     API calls     ┌──────────────────┐
│  Agent  │ ───────────────▶  │  llama.cpp server │
│         │ ◀───────────────  │  (local model)    │
└─────────┘    responses      └──────────────────┘
     │
     ▼
  Your files,
  terminal, etc.

Alternative: llama-agent

llama-agent takes a different approach — it builds the agent loop directly into llama.cpp as a single binary with zero external dependencies. No Node.js, no Python, just compile and run:

git clone https://github.com/gary149/llama-agent.git
cd llama-agent

# Build
cmake -B build
cmake --build build --target llama-agent

# Run (downloads the model automatically)
./build/bin/llama-agent -hf ggml-org/gemma-4-26b-a4b-it-GGUF:Q4_K_M

Because tool calls happen in-process rather than over HTTP, there is no network overhead between the model and the agent. It also supports subagents, MCP servers, and an HTTP API server mode.

Next Steps

Use AI Models Locally — Learn more about running models on your machine
llama.cpp Guide — Detailed llama.cpp installation and usage
Agents on the Hub — Connect agents to the Hugging Face ecosystem

Update on GitHub

←Building agents with the HF SDK Agent Libraries→