- My Forums
- Tiger Rant
- LSU Recruiting
- SEC Rant
- Saints Talk
- Pelicans Talk
- More Sports Board
- Fantasy Sports
- Golf Board
- Soccer Board
- O-T Lounge
- Tech Board
- Home/Garden Board
- Outdoor Board
- Health/Fitness Board
- Movie/TV Board
- Book Board
- Music Board
- Political Talk
- Money Talk
- Fark Board
- Gaming Board
- Travel Board
- Food/Drink Board
- Ticket Exchange
- TD Help Board
Customize My Forums- View All Forums
- Show Left Links
- Topic Sort Options
- Trending Topics
- Recent Topics
- Active Topics
Started By
Message
re: Official LLM discussion thread
Posted on 7/23/26 at 5:36 pm to CAD703X
Posted on 7/23/26 at 5:36 pm to CAD703X
quote:
local language model (?) i think thats what it stands for
Large Language Model.
Running locally is basically free. It will add to electricity bill a little, but the initial investment is pretty pricey. You aren't running anything larger than like a 14b 4 bit quantized model even with a pretty decent rig with with 16 GB of ram and a good integrated graphics card. Also Vram meaning the GPU has a lot higher bandwidth than system ram so it's a lot faster. You won't exactly get the same results running locally as you get from the cloud based servers located in huge data centers unless you have a serious set up locally.
0
Posted on 7/23/26 at 5:54 pm to CAD703X
Just use deepseek 4 flash. It averages 1 cent per million tokens and is multiple times bigger and better than anything you can run on a $5k-10k rig.
I think the most I’ve ever spent on it in a day is about $4 and that was about 400,000,000 tokens.
For frigate, I just use whatever VLM is free on openrouter. I don’t care if they see my butt. It’s just for enhancing semantic search which 1) already has image embeddings 2) I don’t use much at all.
Forgot to answer questions:
I ran: mostly Qwen 3-30b. Hardware: Mostly Nvidia Jetson 32GB. Electricity: almost zero. Do it again?: environment would have to change drastically. Pros: privacy. Cons: literally everything else. Pair with local voice: still run local voice. Bare metal: absolutely. How dovetail into serious LLM: not at all. Tried it: was disastrous. Worst part is I felt like it was OK and didn’t realize how fast I was falling behind. How big of PITA: anything on a Jetson is a PITA, AI adds a lot because things change so fast and to stay on top often means compiling new code constantly.
I think the most I’ve ever spent on it in a day is about $4 and that was about 400,000,000 tokens.
For frigate, I just use whatever VLM is free on openrouter. I don’t care if they see my butt. It’s just for enhancing semantic search which 1) already has image embeddings 2) I don’t use much at all.
Forgot to answer questions:
I ran: mostly Qwen 3-30b. Hardware: Mostly Nvidia Jetson 32GB. Electricity: almost zero. Do it again?: environment would have to change drastically. Pros: privacy. Cons: literally everything else. Pair with local voice: still run local voice. Bare metal: absolutely. How dovetail into serious LLM: not at all. Tried it: was disastrous. Worst part is I felt like it was OK and didn’t realize how fast I was falling behind. How big of PITA: anything on a Jetson is a PITA, AI adds a lot because things change so fast and to stay on top often means compiling new code constantly.
This post was edited on 7/24/26 at 10:43 am
Posted on 7/23/26 at 10:12 pm to CAD703X
Just started thinking about local LLM while pondering which free-level AI agent I'm willing to start paying for. My current prompts are mostly requests for Python scripts both freestanding and for Blender and Photoshop. I also track some science lit.
Gemini tells me that I can do all or most locally if I keep an eye on the model's training cutoff date.
Asking about hardware, Gemini told me to watch: LINK
It's not for a 5 year old, but it's 20-min length lets you get used to "English on the Ganges".
Gemini tells me that I can do all or most locally if I keep an eye on the model's training cutoff date.
Asking about hardware, Gemini told me to watch: LINK
It's not for a 5 year old, but it's 20-min length lets you get used to "English on the Ganges".
This post was edited on 7/23/26 at 10:16 pm
Posted on 7/24/26 at 7:02 am to CAD703X
I have bought next to 0 hardware dedicated to any of my self hosted stuff. It's just refurb/repurposed mostly old tech.
My main LLM server is my former gaming desktop. It has an nvidia RTX 2060 which can run some of the smaller models pretty well. It already runs Linux so I just have ollama in docker with gpu passthrough. There are a few other containers I will also spin up at times. It also serves as a bit of a terminal server. I will put it to sleep when I'm not using it so it's not always on.
I also have an ubuntu LXC running as a guest on a Proxmox server that is an old HP mini prodesk. That one is mostly just used as a fallback when my desktop isn't on since it always is. It's much slower but it's fine for stuff I'm not waiting on a prompt response from.
I have another old desktop that is my "primary" server running Proxmox. It has an Ubuntu server VM that's my docker host. I have openwebui, n8n among other containers.
In all I don't do a huge amount of LLM things, but it's fun to mess around with. You can learn a lot even without big powerful hardware.
My main LLM server is my former gaming desktop. It has an nvidia RTX 2060 which can run some of the smaller models pretty well. It already runs Linux so I just have ollama in docker with gpu passthrough. There are a few other containers I will also spin up at times. It also serves as a bit of a terminal server. I will put it to sleep when I'm not using it so it's not always on.
I also have an ubuntu LXC running as a guest on a Proxmox server that is an old HP mini prodesk. That one is mostly just used as a fallback when my desktop isn't on since it always is. It's much slower but it's fine for stuff I'm not waiting on a prompt response from.
I have another old desktop that is my "primary" server running Proxmox. It has an Ubuntu server VM that's my docker host. I have openwebui, n8n among other containers.
In all I don't do a huge amount of LLM things, but it's fun to mess around with. You can learn a lot even without big powerful hardware.
Posted on 7/24/26 at 10:50 am to CAD703X
I've always hated the webui's for LLM's, recently found a CLI shell that I really like for my local LLM.. allows you to pass the usual prompts, but what I've found far more useful is to pass CLI output to the LLM to be analysed. .sh and .ps1 scripts (robertsinfosec/llm-cli/ on github)
simple prompt example
The above is a result of ansible jobs scheduled to pull playbooks from a git repo to push changes, install software, maintain security settings, user accounts, copy files, populate config's, pretty much anything I want replicated to all of the hosts on my network.
On my primary desktop machine I keep Ollama running in a container with multiple models available depending on what I want to do.
hardware is pretty tame but works quite well.
Ryzen 9 5950X (32 thread)
32G RAM
nvidia 4060ti (16G)
Other instances running in my proxmox lab with various hardware available for other workflows.
simple prompt example
quote:
+--[10:27:26]-[0]-[xxxxxxxxx:/]
+--| tail -1000 /var/log/auth.log | grep opened | llm "scan log for unusual logins"
A security audit!
After reviewing the logs, I've identified some potential issues:
1. **Repeated sudo sessions**: The logs show a large number of repeated `sudo` sessions from `ansible(uid=xxx)` to `root(uid=0)`. This could indicate that someone or something is trying to escalate privileges repeatedly.
2. **Unusual user ID (xxx)**: In the last log entry, you'll notice that the user ID is xxxx, which is not one of the previously seen user IDs (0 for root, xxx for ansible). It's possible that this is a new user or an attacker trying to hide their identity.
3. **Multiple simultaneous login attempts**: There are multiple `sudo` sessions opened in rapid succession by the same user (`ansible(uid=xxx)`). This could indicate brute-force attacks or automated scripts.
To investigate further:
* Review system logs for any suspicious activity around these events.
* Check the sudoers file to ensure that only authorized users can run commands with elevated privileges.
* Investigate why `ansible` is running so many repeated `sudo` sessions. Is this an expected behavior, or could it be a sign of unauthorized access?
* Look into configuring additional security measures, such as rate limiting for login attempts or monitoring tools to detect unusual patterns.
The above is a result of ansible jobs scheduled to pull playbooks from a git repo to push changes, install software, maintain security settings, user accounts, copy files, populate config's, pretty much anything I want replicated to all of the hosts on my network.
On my primary desktop machine I keep Ollama running in a container with multiple models available depending on what I want to do.
hardware is pretty tame but works quite well.
Ryzen 9 5950X (32 thread)
32G RAM
nvidia 4060ti (16G)
Other instances running in my proxmox lab with various hardware available for other workflows.
Posted on 7/24/26 at 2:48 pm to Dallaswho
quote:
Just use deepseek 4 flash. It averages 1 cent per million tokens and is multiple times bigger and better than anything you can run on a $5k-10k rig.
I canceled my Claude sub a few weeks ago and only have a few days left on it. Just got Open WebUI lxc on Proxmox up and running and connected it to deepseek API, so I'm hoping this performs well enough to be my primary AI tool
Posted on 7/24/26 at 3:02 pm to bluebarracuda
Posted on 7/24/26 at 4:15 pm to bluebarracuda
quote:
deepseek API,
I go through openrouter but I hardcode deepseek as my provider.
It’s not Opus, but best thing is cache is basically free. I think 0.25 cents / 1M for flash and only 0.3x cents for pro. So if you’re doing long agentic coding, then the value is insanely good.
Claude subscriptions are a good value but they obviously have to cut back features to make room for higher paying customers.
Ability wise think sonnet or Opus from 3-4 months ago, but maybe a little better at coding and way less general knowledge.
This post was edited on 7/24/26 at 4:25 pm
Posted on 7/24/26 at 4:41 pm to Dallaswho
Yea I really don't need AI for much. Little help here and there with automations and whatnot within HA, optimizing my homelab, stuff like that. I was burning through my limits with claude at first, but now I use it once or twice a week when I have time to sit down with my lab
Posted on 7/26/26 at 8:12 am to CAD703X
Working on this through Windows 11 virtual environment
Ollama
Qwen
Docling
Ocr
Will load through streamlit gui
Made folders that give it a RAG (brain) to pull from documents in upload specifically
Incoming folder documents (pdfs) are analyzed and summarized depending on the script
Output folder is word or Excel format for data review
It's fun and wild to mess with. However I found the analyzing will COMPLETELY make up data. Went through a manual review is a data output and the word document was complete hallucinations
Ollama
Qwen
Docling
Ocr
Will load through streamlit gui
Made folders that give it a RAG (brain) to pull from documents in upload specifically
Incoming folder documents (pdfs) are analyzed and summarized depending on the script
Output folder is word or Excel format for data review
It's fun and wild to mess with. However I found the analyzing will COMPLETELY make up data. Went through a manual review is a data output and the word document was complete hallucinations
This post was edited on 7/26/26 at 8:13 am
Posted on 7/26/26 at 1:28 pm to CAD703X
Is this basically like Jarvis from Ironman? That would be pretty cool to have
Posted on 7/26/26 at 3:31 pm to Yeti_Chaser
I turned the chatgpt sol voice on and my wife walked in on me talking to her and was like wtf who is that woman on the phone 
Guess I better switch to a guy voice. She was actually getting jealous.
Guess I better switch to a guy voice. She was actually getting jealous.
Posted on 7/26/26 at 4:10 pm to CAD703X
Wait can you add a voice on local llm ?
Posted on 7/27/26 at 8:04 am to HailToTheChiz
Yup
Page 2 of 2
Popular
Back to top
Follow TigerDroppings for LSU Football News