Building My Personal AI Homelab: A Hybrid LLM Architecture
AI Homelab — Part 1
AI has quietly become part of almost every technical workflow I run.
Linux and SRE work. Ansible. Terraform. PHP and Laravel. Android development. Documentation. Troubleshooting. Research. Information analysis. Writing. Image generation. Video production.
At this point, I don’t really have an “AI workflow”.
I have an increasingly large collection of AI workflows, each with its own model, subscription, API, limits and pricing model.
And that raises an obvious question:
How much of this actually needs to run in the cloud?
Probably less than I currently send there.
So I’m starting a new homelab project: building a hybrid AI infrastructure on top of my existing Proxmox cluster.
The goal isn’t to build “my own ChatGPT”.
That’s neither realistic nor particularly interesting with the hardware I have.
The goal is much more pragmatic:
Run the workloads that don’t need frontier models locally, route the workloads that do need them to the cloud, and put a proper infrastructure layer in between.
And, because I’m an SRE, I want to be able to measure the result.
The Problem
My current AI stack looks roughly like this:
ChatGPT Claude GitHub Copilot Cloud APIs Image generation Video generation Voice generation
Each tool is good at something.
The problem is that I’m increasingly using expensive cloud inference for workloads that don’t necessarily require it.
For example:
"Explain this systemd error." "Convert this Bash script to Ansible." "Generate this Terraform variable." "Write a regex." "Summarise this log." "Reformat this YAML." "Explain what this Kubernetes manifest is doing."
None of these tasks necessarily require a frontier model.
A reasonably capable local model should be able to handle a significant percentage of them.
And if it can:
local inference
↓
€0 marginal API cost
↓
no data leaving the network
↓
lower latency for some workloads
That’s worth exploring.
The Homelab Reality
I don’t have a dedicated AI server.
I have an existing Proxmox cluster running actual workloads.
The current environment includes things such as:
Frigate DNS n8n Home Assistant
and I don’t intend to starve those workloads just because I’ve become interested in LLMs.
The AI infrastructure therefore has to be a good citizen inside the existing homelab.
The first iteration will run on the node with the most available headroom and will deliberately be conservative with CPU and RAM allocation.
There is also one important constraint:
There is no discrete GPU.
This is a CPU-first experiment.
If the numbers eventually justify adding a GPU, I’ll add one.
But I’m not buying hardware first and trying to justify it afterwards.
The Architecture
The initial architecture looks like this:
LAN
10.10.20.0/24
│
│
┌────────▼────────┐
│ Proxmox │
│ node2 │
└────────┬────────┘
│
┌──────────────┼────────────────┐
│ │ │
▼ ▼ ▼
ai-llm ai-apps ai-infra
10.10.20.20 10.10.20.21 10.10.20.22
│ │ │
│ │ │
Ollama Open WebUI nginx
│ step-ca
LiteLLM Prometheus
cAdvisor Grafana
│
▼
TLS/DNS
The existing n8n instance remains where it is:
LXC 200
I’m not going to deploy another n8n instance just because this is an AI project.
Instead, the existing automation platform becomes one of the components of the new architecture.
Three Initial VMs
The first version is intentionally small.
ai-llm
10.10.20.20
Responsible for:
Ollama Local LLMs
This is where the actual local inference happens.
Initially:
CPU RAM no GPU
I’ll start with relatively small quantized models and benchmark the actual hardware rather than relying on somebody else’s benchmark running on a completely different system.
The interesting metric isn’t:
“How good is model X according to benchmark Y?”
It’s:
“How many useful tokens per second can I get from my actual hardware, and what does that workload do to the rest of the homelab?”
ai-apps
10.10.20.21
This will host the application layer:
Open WebUI LiteLLM cAdvisor
Open WebUI provides the user-facing interface.
LiteLLM becomes the AI gateway.
cAdvisor provides container-level metrics.
ai-infra
10.10.20.22
This VM is the infrastructure layer:
nginx step-ca Prometheus Grafana Node Exporter
The idea is to keep infrastructure services separate from the actual AI workloads.
The AI Gateway
This is probably the most important architectural decision.
I don’t want every application to know how to talk to:
OpenAI Anthropic Google Ollama ...
Instead:
Applications
│
▼
┌────────────┐
│ AI Gateway │
│ LiteLLM │
└─────┬──────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Ollama OpenAI Anthropic
LOCAL CLOUD CLOUD
The application talks to one API.
The gateway decides where the request goes.
That gives me a clean abstraction layer between applications and model providers.
It also gives me somewhere to eventually implement:
routing authentication rate limiting budgets fallbacks logging cost tracking model policies
And potentially:
simple request → local model coding request → Claude reasoning request → frontier model research request → cloud
The exact routing strategy will evolve once I have actual usage data.
*.ai.hferreira.loc
Everything related to this platform will live under:
*.ai.hferreira.loc
For example:
openwebui.ai.hferreira.loc gateway.ai.hferreira.loc grafana.ai.hferreira.loc prometheus.ai.hferreira.loc n8n.ai.hferreira.loc
This is an internal-only namespace.
There is no reason for these services to be exposed directly to the Internet.
The plan is to have a single ingress point:
*.ai.hferreira.loc
│
▼
nginx :443
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Open WebUI Grafana n8n
│
▼
LiteLLM
│
▼
Ollama / Cloud
Internal PKI
There is another piece I want to get right from the beginning:
TLS.
I don’t want the usual homelab experience of:
https://something.internal
followed by:
“Yeah, just click Advanced → Proceed.”
I’m already working towards an internal CA using step-ca, so this project is a good opportunity to put it to use.
The target architecture is:
step-ca
│
▼
Internal certificates
│
▼
*.ai.hferreira.loc
That gives me proper TLS across the internal AI stack without relying on publicly trusted certificates for private services.
It also gives me a proper foundation for service-to-service TLS later.
n8n: The Automation Layer
The existing n8n deployment is going to become a key component of the platform.
Rather than thinking about n8n as “another AI application”, I’m treating it as the workflow orchestration layer.
Conceptually:
Internet
│
┌─────────────┼─────────────┐
│ │ │
BASE Municipal Other
sources sources sources
│ │ │
└─────────────┼─────────────┘
│
▼
n8n
│
▼
AI Gateway
│
┌─────────┴─────────┐
│ │
Ollama Cloud
│ │
└─────────┬─────────┘
▼
processing
│
▼
classification
│
▼
notification
This opens up a much more interesting use case than simply having a chatbot running at home.
For example, I want to eventually build workflows that can:
- monitor public sources;
- retrieve new documents;
- classify them;
- extract relevant information;
- compare them against previous documents;
- summarise changes;
- attach source metadata;
- generate a daily briefing;
- notify me only when something is actually relevant.
That’s where AI starts becoming infrastructure rather than another UI.
Two AI Domains
I also want to keep two completely separate knowledge domains.
Technical
RHEL Linux Ansible AAP Satellite Terraform Kubernetes Azure PHP Laravel Android Bash Python runbooks scripts documentation
Research / Political
municipal documents minutes regulations budgets procurement legislation public datasets news research material
I don’t want these contexts blindly mixed.
A technical RAG should not know about my political research corpus.
A political RAG should not be fed internal infrastructure documentation.
Separation of data domains will therefore be an architectural concern, not just a UI convention.
RAG
Once the basic platform is running, the next step is a proper RAG pipeline.
Something along these lines:
Documents
│
▼
ingestion
│
▼
chunking
│
▼
embeddings
│
▼
Qdrant
│
▼
RAG
│
▼
AI Gateway
│
┌───────┴───────┐
▼ ▼
Ollama Cloud LLM
The important part isn’t just retrieving chunks.
I want the retrieved information to retain provenance:
source document date page URL document type metadata
For research work, the answer isn’t enough.
I need to know:
“Where did this statement come from?”
An LLM hallucinating a plausible answer is not a useful research system.
An LLM producing:
claim → document → page → source URL
is considerably more useful.
Local vs Cloud
I’m not trying to eliminate cloud inference.
That’s not realistic and, in many cases, doesn’t make technical sense.
Instead, I’m looking at a workload hierarchy.
Tier 0 — Local
Ollama
For:
small transformations log analysis YAML JSON regex simple Ansible simple Terraform summarisation rewriting brainstorming
Tier 1 — Developer tooling
GitHub Copilot
For:
autocomplete small refactors inline coding assistance quick fixes
Tier 2 — Cloud frontier models
ChatGPT Claude Gemini
For:
complex reasoning large-context analysis difficult debugging deep research complex code generation
Tier 3 — Specialised AI
video voice image generation
These are workloads where local CPU inference isn’t particularly interesting with my current hardware.
Why CPU First?
Because I want data.
Not opinions.
Not:
“You need a GPU.”
I want to know:
CPU utilisation RAM utilisation tokens/sec latency model size quantisation power consumption cloud fallbacks monthly API cost
Then I’ll have enough information to decide whether a GPU makes sense.
The decision should eventually look like:
GPU cost
↓
expected monthly savings
↓
expected productivity gain
↓
payback period
If that calculation says “buy a GPU”, I’ll buy a GPU.
If it says “keep using the cloud”, I won’t.
Observability From Day One
I don’t want to build an AI black box.
The infrastructure will therefore include:
Prometheus Grafana Node Exporter cAdvisor
The target is to be able to answer questions such as:
How much CPU is Ollama consuming? Which model is being used? How many requests are going local? How many fall back to cloud? What's the average latency? What's the token throughput? How much did the cloud inference cost? What happened to the rest of the Proxmox node while the model was running?
Eventually I’d like a dashboard along these lines:
┌──────────────────────────────────────────┐ │ AI HOMELAB │ ├──────────────────────────────────────────┤ │ Local requests 1,842 │ │ Cloud requests 171 │ │ Local tokens/sec 8.7 │ │ Avg latency 3.2s │ │ CPU 71% │ │ RAM 11.4GB │ ├──────────────────────────────────────────┤ │ Cloud cost this month │ │ │ │ OpenAI €2.41 │ │ Anthropic €1.87 │ │ Gemini €0.74 │ └──────────────────────────────────────────┘
The exact numbers are fictional, obviously.
The dashboard isn’t.
The Economics
This project isn’t about making AI free.
It’s about improving the cost/performance curve.
Suppose a task can be handled locally for:
€0 marginal API cost
and the alternative is:
cloud inference
Then the local option is attractive — assuming the latency and quality are acceptable.
But if a task requires a frontier model and costs a few cents to execute, spending hundreds of euros on hardware to avoid those few cents makes no sense.
The optimisation target is therefore:
maximum useful AI capability per euro.
Not:
maximum number of GPUs per square metre.
What I Expect to Learn
There are several things I’m genuinely curious about.
Can a CPU-only local LLM handle a meaningful percentage of my daily technical workload?
I suspect the answer is yes.
The interesting question is how much.
How much cloud usage can actually be eliminated?
This is where the metrics matter.
Does local RAG provide enough value to justify the operational overhead?
Probably.
But “probably” isn’t a measurement.
Does a GPU make financial sense?
Maybe.
We’ll find out.
How much operational complexity am I willing to accept?
This one is almost guaranteed to become interesting.
Because there is a point where:
saving €10/month
isn’t worth maintaining:
another VM another database another backup another upgrade path another monitoring target
That’s part of the experiment too.
The Real Objective
This isn’t an attempt to replace ChatGPT, Claude, Gemini or any other cloud provider.
It’s an attempt to build a personal AI control plane.
Something that understands:
what workloads I have what data I have what models I have what providers I have what each request costs what can run locally what should stay local what needs the cloud
And then makes those decisions explicit rather than having every application make them independently.
In other words:
AI workloads
│
▼
┌──────────────┐
│ AI Control │
│ Plane │
└──────┬───────┘
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
Local Cloud Specialised
LLMs LLMs services
It’s a very SRE way of looking at AI.
Don’t just consume the service. Build an observable, controllable platform around it.
And then measure whether the platform was actually worth building.
That’s what I’m going to find out.