- My Forums
- Tiger Rant
- LSU Recruiting
- SEC Rant
- Saints Talk
- Pelicans Talk
- More Sports Board
- Fantasy Sports
- Golf Board
- Soccer Board
- O-T Lounge
- Tech Board
- Home/Garden Board
- Outdoor Board
- Health/Fitness Board
- Movie/TV Board
- Book Board
- Music Board
- Political Talk
- Money Talk
- Fark Board
- Gaming Board
- Travel Board
- Food/Drink Board
- Ticket Exchange
- TD Help Board
Customize My Forums- View All Forums
- Show Left Links
- Topic Sort Options
- Trending Topics
- Recent Topics
- Active Topics
Started By
Message
Any "homelabbers" want to share their setups?
Posted on 2/14/21 at 12:25 pm
Posted on 2/14/21 at 12:25 pm
I feel like a terrible geek because I do not yet have a solid setup for the various servers and services that I run and want to run. Right now I just have an old-ish desktop machine that I use as a server. It runs Plex server, various media acquisition services, my unifi controller, zoneminder server, basic data storage, and various other things.
What I want is to start using VMs and containerized services to make everything more stable and organized. Problem is I don't have much experience with this stuff, and I'm a little confused about the best way to set this up from scratch.
I think I have settled on Proxmox for VMs, RancherOS for Docker container management, and FreeNAS for storage. Good choices?
I'm not sure how to "stack" all of this in either hardware or software. I'm definitely going to switch to rack mount hardware, but should I get two servers? One for Proxmox, and another for storage with FreeNAS? I believe FreeNAS will run on Proxmox, but I think that introduces special considerations and complexities, so I guess it would be cleaner to separate my storage from my virtualization with two machines?
So I imagine as follows: 1U server running proxmox with RancherOS in a VM and my various services running in docker containers, and maybe a 2U hot-swap chassis running FreeNAS on bare metal. I should be able to store my VMs and docker containers on FreeNAS, and Proxmox will load them up just fine right?
Am I on the right track?
What are some thing you love or hate about your setup? Are there any must-do's or must-don't's?
What I want is to start using VMs and containerized services to make everything more stable and organized. Problem is I don't have much experience with this stuff, and I'm a little confused about the best way to set this up from scratch.
I think I have settled on Proxmox for VMs, RancherOS for Docker container management, and FreeNAS for storage. Good choices?
I'm not sure how to "stack" all of this in either hardware or software. I'm definitely going to switch to rack mount hardware, but should I get two servers? One for Proxmox, and another for storage with FreeNAS? I believe FreeNAS will run on Proxmox, but I think that introduces special considerations and complexities, so I guess it would be cleaner to separate my storage from my virtualization with two machines?
So I imagine as follows: 1U server running proxmox with RancherOS in a VM and my various services running in docker containers, and maybe a 2U hot-swap chassis running FreeNAS on bare metal. I should be able to store my VMs and docker containers on FreeNAS, and Proxmox will load them up just fine right?
Am I on the right track?
What are some thing you love or hate about your setup? Are there any must-do's or must-don't's?
5
Posted on 2/14/21 at 1:57 pm to Korkstand
I've got a dual Xeon 2660 v2 server with 128gb of ram in a silverstone hotswap server case. I've got the 5.25 bays filled with a 2.5" hotswap caddy too.
I have my 42u server rack in storage at my parents since there's no space in my house for it. Whenever I move outta this nola shotgun I'm going to get my rack mount back setup
I have my 42u server rack in storage at my parents since there's no space in my house for it. Whenever I move outta this nola shotgun I'm going to get my rack mount back setup
Posted on 2/14/21 at 2:10 pm to bluebarracuda
What are you running on that?
Posted on 2/14/21 at 2:19 pm to Korkstand
Proxmox. Plex, some game servers, openvpn, file server
Posted on 2/14/21 at 2:30 pm to bluebarracuda
What are you using as file server?
Posted on 2/14/21 at 5:21 pm to Korkstand
- OpenBSD on some Zotac nano PC with dual NICs as a router/networking services device
- Ubuntu server on a Ryzen 2700 machine with 64GB RAM as a home server
- Aruba Networks enterprise switch that has four 10G ports
- Unifi hotspot
Ubuntu server uses ZFS and has four 4TB drives in raidz1 (this holds audio/video files for Plex) and six 6TB drives in raidz2 (for the rest of storage needs for my household).
My server, workstation, and wife's PC have 10G NIC cards.
The Ubuntu server is just in a regular ATX case (Fractal R5) but I'm planning on putting everything in my office closet in a rack mount setup in the future.
For containerization, at this point I'm just running Ubuntu on bare metal and using Docker for services like Plex, Transmission (with VPN), nextcloud, etc. I personally feel it's more complicated to run another OS like FreeNAS, RancherOS for each service so I don't bother with all of that. Maybe if I had a more needs it might make sense, but most of this is just stuff for my wife and I to use.
- Ubuntu server on a Ryzen 2700 machine with 64GB RAM as a home server
- Aruba Networks enterprise switch that has four 10G ports
- Unifi hotspot
Ubuntu server uses ZFS and has four 4TB drives in raidz1 (this holds audio/video files for Plex) and six 6TB drives in raidz2 (for the rest of storage needs for my household).
My server, workstation, and wife's PC have 10G NIC cards.
The Ubuntu server is just in a regular ATX case (Fractal R5) but I'm planning on putting everything in my office closet in a rack mount setup in the future.
For containerization, at this point I'm just running Ubuntu on bare metal and using Docker for services like Plex, Transmission (with VPN), nextcloud, etc. I personally feel it's more complicated to run another OS like FreeNAS, RancherOS for each service so I don't bother with all of that. Maybe if I had a more needs it might make sense, but most of this is just stuff for my wife and I to use.
Posted on 2/14/21 at 5:44 pm to Korkstand
Watching this thread. I'm digging into CCNP SP certification and want to broaden my IT knowledge base aside from networking. Considering a server for virtualized router labbing, scripting practice, learning vmware, containerization, and hybrid cloud over time. I know you can get a 1RU Dell poweredge r720 for ~$300-350 and that should suit needs, but want to see what else may be out there.
This post was edited on 2/14/21 at 6:32 pm
Posted on 2/14/21 at 8:23 pm to efrad
quote:If I understand correctly (and maybe I don't), RancherOS will be running in a Proxmox VM, and then all of my containerized services will run in that one RancherOS VM (and not a whole new VM per service).
For containerization, at this point I'm just running Ubuntu on bare metal and using Docker for services like Plex, Transmission (with VPN), nextcloud, etc. I personally feel it's more complicated to run another OS like FreeNAS, RancherOS for each service so I don't bother with all of that.
Then I was thinking I'd run FreeNAS on its own hardware.
Posted on 2/14/21 at 8:58 pm to Korkstand
Dolphin talkers nerdin' out like a mf'er in here.
Posted on 2/15/21 at 6:20 am to GrizzlyAlloy
My lab -
Esxi cluster consists of;
3 X AMD FX 8350 (8 core) / 32G RAM
2 X Dual CPU Celeron 4 cores / 32G RAM
2 X RaspiPi4's (1-8G and 1-4G) running ARM esxi
(yes, esx runs GREAT on the Pi's!)
File servers - -
Xpenology - 4 core Intel 16G RAM (44TB)
FreeNAS - 6 core AMD FX 8320 32G
3 Workstations are all Ryzen5 3600X 32G w/2TB m.2
and a whole bunch of others running as needed.
Esxi cluster consists of;
3 X AMD FX 8350 (8 core) / 32G RAM
2 X Dual CPU Celeron 4 cores / 32G RAM
2 X RaspiPi4's (1-8G and 1-4G) running ARM esxi
(yes, esx runs GREAT on the Pi's!)
File servers - -
Xpenology - 4 core Intel 16G RAM (44TB)
FreeNAS - 6 core AMD FX 8320 32G
3 Workstations are all Ryzen5 3600X 32G w/2TB m.2
and a whole bunch of others running as needed.
This post was edited on 2/15/21 at 6:43 am
Posted on 2/15/21 at 10:59 am to dakarx
Now this is some real homelabbin' shite. 
So your RPi's are part of the same esxi cluster as the others? How does that work out?
So your RPi's are part of the same esxi cluster as the others? How does that work out?
Posted on 2/15/21 at 12:06 pm to Korkstand
My lab is pretty much designed from JDM_WAAAT’s guides on his website, Serverbuilds.net. Still pretty new, expanded from just the Mac Mini running plex on external SSDs to the whole build below this past year, but have everything working where I want it now. Also finally ran cat 6 to the TVs around the house for reliable 4k streaming, and upped my transcoding game for outside the network using QuickSync.
1 HP290 with an added dual Intel NIC/NVME for pfSense
1 HP290 with an added i5-9400k, 1tb NVME, ram on Ubuntu dedicated for Plex - beefed this one up for transcoding
1 ProDesk 400 with an added i5-7600/1tb NVME running Ubuntu, with Docker/Rancher for Deluge, Sonarr, Radar and a few other services
A Naskiller 4.0 build running Unraid with 30tb of usable space with parity
1 late 2014 MacMini with an added SSD for Mac to Linux automation (mainly iCloud to server)
1 Raspberry Pi 4 running HOOBs
1 Raspberry Pi 3 running Pi-Hole
The itch was scratched - now need to find a couple of projects for some RP Zero-W’s I’ve got floating around.
1 HP290 with an added dual Intel NIC/NVME for pfSense
1 HP290 with an added i5-9400k, 1tb NVME, ram on Ubuntu dedicated for Plex - beefed this one up for transcoding
1 ProDesk 400 with an added i5-7600/1tb NVME running Ubuntu, with Docker/Rancher for Deluge, Sonarr, Radar and a few other services
A Naskiller 4.0 build running Unraid with 30tb of usable space with parity
1 late 2014 MacMini with an added SSD for Mac to Linux automation (mainly iCloud to server)
1 Raspberry Pi 4 running HOOBs
1 Raspberry Pi 3 running Pi-Hole
The itch was scratched - now need to find a couple of projects for some RP Zero-W’s I’ve got floating around.
This post was edited on 2/15/21 at 12:08 pm
Posted on 2/15/21 at 12:11 pm to Korkstand
quote:
If I understand correctly (and maybe I don't), RancherOS will be running in a Proxmox VM, and then all of my containerized services will run in that one RancherOS VM (and not a whole new VM per service).
I’m no expert either - but you are correct here. Container in a container. Nesting dolls of containers. One thing to think about is if you run plex in a VM what kind of hardware acceleration you’ll attribute to it - some gfx cards have problems allocating to the correct virtualization without some work.
Posted on 2/15/21 at 4:17 pm to Korkstand
Start small and go from there. Ebay is your best friend. Or local pickups if you live in a large town or near a tech area. You can get a LOT of good deals for "e-waste". Power consumption was/is my concern. Due to that, I have decommissioned a few of my servers as it was overkill. Still use them if I'm doing contract work for companies. I built in a power meter to my 3000 watt ups for $10 if you're OK with wiring it yourself. Just sharing my exp:
LINK
I have since replaced the mellanox 40gbe with a standard 16 port sfp+ fiber switch, where I'm running that everywhere (put in dual solarflare 10g cards in each box).
Added another DL580 in another rack.
My network is a Firebox xtm800 series (with PFSense) to an aruba S3500 in a smaller network rack (where my fiber is mounted as well).
I run esxi 6.5 on 2 of my servers (DL360\DL380), but my nas is a Debian build. I hand-made my OS with stuff I wanted as I couldn't get openmediavault\freenas things to work with the hardware. My nas is running a 3900x with 64g of ram. 500g nvme for transcoding\OS. 35TB in various ZFS pools.
I also run a blueiris VM for my cameras which mounts my NAS drives over fiber to store the videos for archiving.
Running various small VMs, pihole, a few containers, Plex/Samba/Cifs on my NAS along with Nextcloud for my own cloud storage.
Snort\IDS on the firebox, along with a LAG'd 4 port connection from the firebox to the aruba. 2 LAG'd sfp+ trunk lines between that and another poweredge 5000 series.
Did all of this for fun over a long period of time. All repurposed Ebay buys except for the fiber switch. Total I paid over the years has been around $3.5k (minus the NAS hardware as that used to be my desktop before upgrading). The hard drives where the most expensive by far.
Edited: typos on model numbers.
LINK
I have since replaced the mellanox 40gbe with a standard 16 port sfp+ fiber switch, where I'm running that everywhere (put in dual solarflare 10g cards in each box).
Added another DL580 in another rack.
My network is a Firebox xtm800 series (with PFSense) to an aruba S3500 in a smaller network rack (where my fiber is mounted as well).
I run esxi 6.5 on 2 of my servers (DL360\DL380), but my nas is a Debian build. I hand-made my OS with stuff I wanted as I couldn't get openmediavault\freenas things to work with the hardware. My nas is running a 3900x with 64g of ram. 500g nvme for transcoding\OS. 35TB in various ZFS pools.
I also run a blueiris VM for my cameras which mounts my NAS drives over fiber to store the videos for archiving.
Running various small VMs, pihole, a few containers, Plex/Samba/Cifs on my NAS along with Nextcloud for my own cloud storage.
Snort\IDS on the firebox, along with a LAG'd 4 port connection from the firebox to the aruba. 2 LAG'd sfp+ trunk lines between that and another poweredge 5000 series.
Did all of this for fun over a long period of time. All repurposed Ebay buys except for the fiber switch. Total I paid over the years has been around $3.5k (minus the NAS hardware as that used to be my desktop before upgrading). The hard drives where the most expensive by far.
Edited: typos on model numbers.
This post was edited on 2/16/21 at 7:56 pm
Posted on 2/15/21 at 4:40 pm to Korkstand
quote:
So your RPi's are part of the same esxi cluster as the others? How does that work out?
Yup, mixed in with the rest no issues. vCenter complained until I added a real license to the ARM vSphere instance...(the free license doesn't seem to care what architecture). Compatibility checks work preventing from migrating compute resources between x86 and ARM, although I did try just hoping for the carnage.
Overall performance of ARM VM's is surprisingly good.
Posted on 2/15/21 at 4:41 pm to g650Tide
Better take down that link, porn is not allowed on TD!
Posted on 2/16/21 at 6:31 am to g650Tide
Now THAT is a home lab.
Posted on 2/16/21 at 8:57 am to g650Tide
quote:
g650Tide
Nice setup! How bad is the noise level?
I had the chance to pick up a dozen Dell 720's refurbs for pretty cheap (cab and all), but was afraid the noise level would have driven Mrs. to flatten my skull.
Posted on 2/16/21 at 2:49 pm to dakarx
Thanks. It's not bad to be honest, but it's also an icebox in my room (my computer room is 10x10, but I have a 13000 btu window unit due to heat) so the window unit drowns everything out. This room is dedicated to all my tech so noise wasn't a concern to start with. Without the window unit it gets HOT and all the fans kick up. So yeah, it would be pretty noisy if in say, in a livable room.
Also here is my network setup I mentioned. I have since replaced the firebox with a newer version. The blue cables are my security cameras, and the black ones are my LAG. Messy wiring, still need to fix that.
LINK
Also here is my network setup I mentioned. I have since replaced the firebox with a newer version. The blue cables are my security cameras, and the black ones are my LAG. Messy wiring, still need to fix that.
LINK
This post was edited on 2/16/21 at 7:47 pm
Posted on 6/12/21 at 1:14 pm to g650Tide
Update: Scored a deal on a used 42u cabinet, and ML350-G6, and a couple Aruba 1G POE switches (24 & 48 port), and some other odds and ends for a couple hundred bucks. Added a couple more servers, an R620, 3 R610, and an NR12000. Migrating everything over to 10G now. The little MikroTik 10G switch/router really screams once properly trunked! 1u KVM 18" display, and 1U 16 port KVM switch.
New pic is NSFW - Not Safe for WIFE (if she keeps the checkbook)
New pic is NSFW - Not Safe for WIFE (if she keeps the checkbook)
Page 1 of 2
Popular
Back to top
Follow TigerDroppings for LSU Football News