Run Llama on a $12 VPS: Local AI Without the GPU Bill

By /

Your Private ChatGPT for the Price of Lunch

Here’s a party trick that never gets old: SSH into a $12/month VPS, type ollama run llama3.2, and ask it something. No account, no API key, no per-token billing, no “as an AI language model” lecture about your data. Just a large language model running on a server you control.

You don’t need a GPU. You don’t need a $96/month monster instance. A budget VPS with enough RAM runs the smaller Llama models just fine for personal use — drafting, summarizing logs, rubber-duck debugging at 2 AM.

This guide walks through the whole thing: picking the VPS, installing Ollama, choosing a model that fits your RAM, and setting expectations honestly.

What You Actually Need (It’s RAM, Not CPU)

LLM inference on CPU is memory-bound, not compute-bound. The model weights have to live in RAM, and that’s the whole game. A rough rule:

  • 2 GB RAM: 1B-parameter models (Llama 3.2 1B). Toy-grade but real.
  • 4 GB RAM: 3B-parameter models (Llama 3.2 3B). Genuinely useful for chat, summarization, and simple Q&A.
  • 8 GB RAM: 8B-parameter models (Llama 3.1 8B). The sweet spot — noticeably smarter.
  • 16 GB+ RAM: 14B–32B models. Now you’re cooking, but you’re out of $12 territory.

These assume quantized models (Q4_K_M, Ollama’s default), which shrink weights to ~4 bits with minimal quality loss. An 8B model at Q4 is about 4.9 GB on disk and in RAM. The math is simple: parameters × 0.5 bytes ≈ RAM needed, plus ~1 GB for the OS and Ollama itself.

If you haven’t picked a VPS yet, read how to choose a VPS first — the short version is: prioritize RAM, then CPU, and don’t overthink the rest for this workload.

Step 1: Spin Up the VPS

Any mainstream distro works. Ubuntu 24.04 LTS or Debian 12, 64-bit. You want:

  • 2 vCPU (4 is nicer, but 2 works)
  • 4 GB RAM minimum (8 GB if you want the 8B model)
  • 25+ GB SSD (models are 2–5 GB each; you’ll collect them like Pokémon)

Standard sysadmin hygiene applies: SSH keys, a firewall allowing only 22 (and whatever else you need), automatic security updates. Nothing AI-specific about this part.

Step 2: Install Ollama

Ollama is the easiest on-ramp — it handles model downloads, quantization, the inference server, and an OpenAI-compatible API in one binary:

curl -fsSL https://ollama.com/install.sh | sh

Verify it’s running:

ollama --version
systemctl status ollama

The installer sets up a systemd service that starts on boot. The API listens on localhost:11434 by default — which is exactly where you want it until you deliberately expose it (more on that in the self-hosting guide).

Step 3: Pull a Model and Run It

ollama pull llama3.2:3b
ollama run llama3.2:3b

That’s it. You’re chatting with Llama 3.2 3B. Try asking it to explain a chunk of an nginx config or summarize a log file — paste text right into the prompt.

Other models worth trying on a budget box:

  • llama3.2:1b — fastest, weakest. Good for testing the pipeline.
  • qwen2.5:3b — strong alternative to Llama 3.2 3B, better at some coding tasks.
  • llama3.1:8b — needs 8 GB RAM, but the quality jump is real.

Browse the full library at ollama.com/library — every model page lists its sizes and RAM requirements.

Step 4: Set Honest Expectations

On 2–4 shared vCPUs, expect roughly 5–15 tokens per second for a 3B model. That’s slower than ChatGPT but perfectly usable for interactive chat — about reading speed. The 8B model on 4 vCPUs runs closer to 3–8 tokens/sec: fine for async tasks, slightly tedious for back-and-forth.

Two knobs that matter:

  • Context length (num_ctx): longer context eats RAM fast. The default 4K is fine; bumping to 8K+ on a 4 GB box invites the OOM killer.
  • Keep-alive: by default Ollama unloads models after 5 minutes idle, so the first prompt after idle pays load time (~10–30s from SSD). Set OLLAMA_KEEP_ALIVE=24h if you have RAM to spare and hate waiting.
# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_KEEP_ALIVE=24h"

Making It Actually Useful

The chat CLI is a demo; the API is the product. Ollama speaks the OpenAI chat-completions dialect, so anything that talks to OpenAI can talk to your VPS:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Summarize these log lines: ..."}]}'

Pipe scripts into it. Summarize cron output. Draft incident reports. It won’t replace a frontier model for hard reasoning, but for the 80% of “read this and summarize” work, a local 3B model is shockingly adequate — and it never sends your logs to anyone.

The Money Math

A $12/month VPS running 24/7 is ~$144/year for unlimited inference. The equivalent in API calls depends on volume, but if you’re piping logs and docs through it daily, the VPS pays for itself fast — and the price is flat whether you ask it one question or ten thousand.

The real cost is your time setting it up, which this guide just reduced to about 20 minutes. The self-hosting guide covers the next steps: exposing the API securely, adding a web UI, and putting it behind a reverse proxy.

Scroll to Top