Official LLM discussion thread

Novice here…what is the benefit of running an LLM locally? What does it achieve versus accessing LLM through the cloud?

quote:

---

Novice here…what is the benefit of running an LLM locally? What does it achieve versus accessing LLM through the cloud?

---

Most of the same benefits of running anything locally. Same cons as well.

Mostly boils down to data ownership and control.

Honestly this is a question I'd send to a cloud LLM.

Ubuntu/ollama 16gb card giving ollama all the access it wants.

Dell 5820 128gb ram 1tb ssd


Building containers to satisfy everything else. I found the speed loss through proxmox was noticeable so that’s why I pulled it.

You can get way more uncensored models with local llm.

edited: Adding the use of cloudflare free tunnel to create an outbound connection only to cloudflare that you can use for remote access.

LINK

quote:

---

Building containers to satisfy everything else. I found the speed loss through proxmox was noticeable so that’s why I pulled it.

---

explain this architecture a bit to me. interested in all options since i haven't begun to provision the PC yet.

That mac had 512 GB of unified memory. You can put a cutting edge 4-bit model on it.

quote:

---

I already have the hardware to play factorio, so free to me.

---

from what i understand; you need low power GPUs for AI/LLMs that can spin up on demand and gaming GPUs are typically higher power draw all the time and more suitable as game engines but again, i know just enough about this stuff to sound like an idiot

Let me know if you want more specifics, but the mgmt was just way easier through base ubuntu as opposed to PM vms. I liked learning and messing with PM but now i was managing a hypervisor on top of other things.

Pulled my LLM stuff off Proxmox and moved to bare-metal Ubuntu with a GPU passthrough-free setup, then containerized everything else around it. Here's the rundown:

**Hardware/OS**
- Bare Ubuntu install (no hypervisor) on a Xeon box, dedicated GPU for local LLM inference
- Reasoning: skipping the VM layer meant the GPU was just... there, no passthrough config, no driver headaches inside a guest

**Container stack (all Docker Compose, managed through Dockge)**
- Open WebUI in front of local models (Ollama) — this is the "LLM out of Proxmox" piece
- Media: Jellyfin, Navidrome, Beets for tagging
- Photos: Immich (server + ML + Postgres/pgvector + Redis)
- Docs/wiki: Wiki.js (Postgres-backed)
- CRM: EspoCRM (MariaDB-backed)
- Offline knowledge: Kiwix serving Wikipedia/Khan Academy ZIMs
- Dashboards: Homepage + Homarr as front doors, Glances for system monitoring
- Filebrowser for poking around the NAS mounts from a browser
- SearXNG for private search
- All NAS storage comes in over CIFS mounts from a separate box, with a systemd override so Docker waits for the mounts before starting containers (otherwise anything touching `/mnt/*` fails hard on reboot)

**Local DNS + routing**
- Avahi for mDNS so everything resolves as `service.local` on the LAN
- One nginx container doing simple 301 redirects from `service.local` ? `ip:port` per container, so nobody has to remember port numbers

**Remote access**
- Cloudflare Tunnel (`cloudflared`) — outbound-only connection from the box to Cloudflare's edge, so there's nothing listening on the router at all. One tunnel, one config file, a hostname-per-service ingress list.
- Cloudflare Access sits in front of the whole wildcard domain and requires login (email one-time-code) before any request even reaches the tunnel — so even the services with zero built-in auth (looking at you, Glances) aren't just sitting open on the internet.
- SSH rides the same tunnel as a raw TCP service, wrapped through `cloudflared access ssh` client-side instead of opening port 22 externally.

**Backups**
- Monthly cron job tars up all the stack config directories + relevant Docker volumes, keeps the last 6, prunes the rest

Happy to go deeper on any piece of this if useful — the tunnel/Access setup in particular was the part that took the most care to get right (mostly making sure nothing was reachable *before* the auth policy was actually in place, not after).


Forgot to add you will need your own domain name to pull off the cloudflare piece. Those are pretty cheap and easy these days though.

Is that power cost 24/7 usage on the LLM though?

I tried to roughly estimate 24/7 always-on with a couple hours of active LLM use per day. If you're idling at 50 watts that'll burn 438kWh/year which is probably $50 minimum anywhere. Maybe you can tune idle consumption lower than that.