Large Language Model.

Running locally is basically free. It will add to electricity bill a little, but the initial investment is pretty pricey. You aren't running anything larger than like a 14b 4 bit quantized model even with a pretty decent rig with with 16 GB of ram and a good integrated graphics card. Also Vram meaning the GPU has a lot higher bandwidth than system ram so it's a lot faster. You won't exactly get the same results running locally as you get from the cloud based servers located in huge data centers unless you have a serious set up locally.

0 ...

Report Post

Posted by Dallaswho

Texas

Member since Dec 2023

3688 posts

Posted on 7/23/26 at 5:54 pm to CAD703X

Just use deepseek 4 flash. It averages 1 cent per million tokens and is multiple times bigger and better than anything you can run on a $5k-10k rig.

I think the most I’ve ever spent on it in a day is about $4 and that was about 400,000,000 tokens.

For frigate, I just use whatever VLM is free on openrouter. I don’t care if they see my butt. It’s just for enhancing semantic search which 1) already has image embeddings 2) I don’t use much at all.

This post was edited on 7/23/26 at 6:00 pm

0 ...

Report Post

Posted by Tree_Fall

Member since Mar 2021

1308 posts

Posted on 7/23/26 at 10:12 pm to CAD703X

Just started thinking about local LLM while pondering which free-level AI agent I'm willing to start paying for. My current prompts are mostly requests for Python scripts both freestanding and for Blender and Photoshop. I also track some science lit.

Gemini tells me that I can do all or most locally if I keep an eye on the model's training cutoff date.

Asking about hardware, Gemini told me to watch: LINK

It's not for a 5 year old, but it's 20-min length lets you get used to "English on the Ganges".

This post was edited on 7/23/26 at 10:16 pm

Page 1 2