Official LLM discussion thread

vllm with qen3.6

explain this to me like i'm a 5 year old

quote:

---

explain this to me like i'm a 5 year old

---

local language model (?) i think thats what it stands for.

basically you're taking 'chatgpt' (the opensource stuff anyway) and installing it on your local server so you can interact with a 'chatbot' that lives with you as opposed to the cloud avoiding
* forced changes and updates that break what you working on
* paywall changes and price increases
* giving Big Tech insight into everything you are asking about.

for me, i can imagine everything from 'learning' things about you and your family based on interactions to make relevant gags or observations "CAD left the wet clothes in the washing machine AGAIN" or if you go with local voice control, it can expand from simple "turn light x on/off" to something more nuanced like “What happened while I was gone?” or “Why is the house using so much electricity?” or “Summarize anything unusual today.”

im a hack and basically starting from ground zero on this stuff so some of the smarter posters on here can weigh in with better use cases but it never hurts to think about doing things like this and less dependent on our benevolent tech overlords to always have our best interests in mind.

eta this is most relevant for new 'home labbers' who are spec-ing out servers to plan on purchasing one with a GPU that is capable of handling some modest LLM as well. it never hurts to think ahead.

It's free

quote:

---

It's free

---

Minus the hardware.

I was looking into this yesterday. I'd like to get a Mac Studio m3 Ultra, but they stopped production and they were about $7,000 at that point I think. That's 5.8 years of Claude Max subscriptions (and no maintenance). I'm gonna wait until they can get the quantization down without losing accuracy and hardware costs come down, hopefully.

What is Quantization?
It is essentially file compression for AI models. Under the hood, an AI is just a massive spreadsheet of highly precise numbers. Quantization rounds those numbers down to take up far less space. Think of it like taking a massive, uncompressed RAW photo and saving it as a smaller JPEG.

Why is it done?
By default, powerful AI models are so huge they require expensive data center supercomputers just to turn on. Quantization shrinks the model's footprint by up to 75% so it can run directly on everyday consumer hardware—like a standard PC, laptop, or smartphone—without noticeably losing its "smartness."

quote:

---

I was looking into this yesterday. I'd like to get a Mac Studio m3 Ultra, but they stopped production and they were about $7,000 at that point I think. That's 5.8 years of Claude Max subscriptions. I'm gonna wait until they can get the quantization down without losing accuracy and hardware costs come down.

---

damn baw

i'm looking at an i7 dell 3660 with maybe a 20 gb RTX 4000 SFF Ada for a fraction of that cost.

quote:

---

Minus the hardware.

---

I already have the hardware to play factorio, so free to me.

for 24/7 use yea, it's a waste of power. I generally use mine for agentic coding, so I can boot everything up with the right OS, run my tasks from laptop for a few hours or overnight, and switch my gaming pc back to windows gaming.

I use n8n for automation and it can run local llm's but they are very slow and not very good for a low power pc, and I've not had the best results using it in those workflows.

quote:

---

quote:

---

It's free

---

Minus the hardware.

---

And the power.

My math says a modest machine and modest usage would cost ~$100/year to run at 15 cents/kwh.

quote:

---

local language model (?)

---

Large language model

Yours would be called local LLM