Psychos little HomeLab

[PsychoWards]

I was in the midst of putting together my new server, which was supposed to run TrueNas, when I received the e-mail that Hexos was available and that's how I ended up here. 🙂

This server was supposed to replace my 10 year old NAS and to replace my current Windows Server running docker and some game servers when needed.

The HW specs are:

CPU: Intel i5 14600K with Thermal Grizzly PhaseSheet PTM (Phase Shifting Pad will not try up compared to thermal paste, thus will not require maintenance in the future)
CPU-Cooler: Noctua NH-D9L
Memory: G.Skill Ripjaws S5 128GB DDR5-5600 (4x32GB)
MoBo: ASRock Z790 PRO RS
PSU: Corsair RM650
NIC: QSFPTEK 10G SFP+ (2x Intel X520-DA2)
Boot Drives: 2x WD Red SN700 500GB
HDDs: 6x Seagate IronWolf 4TB
SSDs: 2x Samsung 990 Evo 1TB
SSD cooler: 3x be quiet! MC1 Pro (since the MoBo only has 1 m.2 slot with a cooler)
external HDD: 1x Seagate Desktop Drive 8TB
case: Inter-Tech IPC 4U-4129L
UPS: APC BX1600MI

I didn't want to get 10 year old used server grade HW, I wanted something newer, faster and more energy efficient, thus the HW above. Yes I know that it would have been better to gett ECC Memory, but ECC Memory and new consumer grade HW is an absolute nightmare, but at least DDR5 got some On-Die EEC which is better then nothing. Neither my old Synology NAS, nor my Unraid Server are rocking ECC memory, so i'm taking my chances.

The server is running Proxmox with one VM running Hexos and another running Ubuntu Server and it's configured to shutdown once the UPS is below 30% of battery power. Both NICs are configured as a Bond in active backup mode, meaning if 1 fails, the other will automagically take over without any interrupt of data transfer (I did test this, it's working great 🙂 ). Both of the VMs are setup with UEFi instead of BIOS and as Q35 Machines instead of the default i440fx. (because PCI-e passthrough requires Q35 and I just wanted to be safe) I'm planning on setting up a 3. VM running Windows in case I need it for some game server or if I want to quickly deploy something (my windows skill are far superior compared to my linux skills)

The Hexos VM has 6 Cores and 64GB of Memory, 1 RaidZ2 Pool with the 6 HDDs, 1 Mirrored Pool with the SSDs and the external HDD will be used for replication of the most important files (pictures and documents). This last part however is still WIP.
So far I've setup periodic SMART tests, Scrub Tasks and periodic Snapshot Tasks. I've also migrated most of my data at this point.

The Ubuntu VM got access to all cores and 48GB of Memory and Docker is running:

• Home Assitant
• Vaultwarden
• Portainer
• Cloudflare-DDNS
• Heimdall
• Nginx Proxy Manager
• Watchtower
• Obico
• Navidrome (which I still need to setup on the new server, this is currently still running on the old server)

Planned to add:

• Jellyfin (not sure if I'm ever gonna use it though, so far I've just mounted the NAS Folder via NFS in Kodi on my Shield TV and never had any issues, I'm not planning on streaming outside of my LAN)
• Sth. do manage Documents/PDF (if anyone got some recommendations please but them down below)
• Immich
• Tailscale (maybe?)
• ???

Currently the Ubuntu OS and docker containers are running of the boot drive, my goal is however, to have everything related to docker running of the Hexos SSD pool and to take periodic images of the VM and backup them to the HDD pool.

I've recently bought a Topton N100 which is now running OPNsense (and used as a Router) and I need to get PiHole working again, but I'll try to get it running on the N100. While changing my router I also changed my network range, meaning I'm no longer able to access Hexos until this bug is fixed. This is mostly fine, since I've been using it to manage access rights and I've got most things set up as I need them. (god damn, access managed is a true nightmare in truescale...)

I've an Unraid Server which I will use to replicate everything from Truenas and last but not least, my old NAS will be used as an off side Backup in case the building burns down. 🙃

That's all from my side, if you have any questions, remarks or recommendations please leave them below. Also if anyone knows where to buy good & cheap open server racks in europe please let me know.

[Zizuar]

13 hours ago, PsychoWards said:

I was in the midst of putting together my new server, which was supposed to run TrueNas, when I received the e-mail that Hexos was available and that's how I ended up here. 🙂

This server was supposed to replace my 10 year old NAS and to replace my current Windows Server running docker and some game servers when needed.

The HW specs are:

CPU: Intel i5 14600K with Thermal Grizzly PhaseSheet PTM (Phase Shifting Pad will not try up compared to thermal paste, thus will not require maintenance in the future)
CPU-Cooler: Noctua NH-D9L
Memory: G.Skill Ripjaws S5 128GB DDR5-5600 (4x32GB)
MoBo: ASRock Z790 PRO RS
PSU: Corsair RM650
NIC: QSFPTEK 10G SFP+ (2x Intel X520-DA2)
Boot Drives: 2x WD Red SN700 500GB
HDDs: 6x Seagate IronWolf 4TB
SSDs: 2x Samsung 990 Evo 1TB
SSD cooler: 3x be quiet! MC1 Pro (since the MoBo only has 1 m.2 slot with a cooler)
external HDD: 1x Seagate Desktop Drive 8TB
case: Inter-Tech IPC 4U-4129L
UPS: APC BX1600MI

I didn't want to get 10 year old used server grade HW, I wanted something newer, faster and more energy efficient, thus the HW above. Yes I know that it would have been better to gett ECC Memory, but ECC Memory and new consumer grade HW is an absolute nightmare, but at least DDR5 got some On-Die EEC which is better then nothing. Neither my old Synology NAS, nor my Unraid Server are rocking ECC memory, so i'm taking my chances.

The server is running Proxmox with one VM running Hexos and another running Ubuntu Server and it's configured to shutdown once the UPS is below 30% of battery power. Both NICs are configured as a Bond in active backup mode, meaning if 1 fails, the other will automagically take over without any interrupt of data transfer (I did test this, it's working great 🙂 ). Both of the VMs are setup with UEFi instead of BIOS and as Q35 Machines instead of the default i440fx. (because PCI-e passthrough requires Q35 and I just wanted to be safe) I'm planning on setting up a 3. VM running Windows in case I need it for some game server or if I want to quickly deploy something (my windows skill are far superior compared to my linux skills)

The Hexos VM has 6 Cores and 64GB of Memory, 1 RaidZ2 Pool with the 6 HDDs, 1 Mirrored Pool with the SSDs and the external HDD will be used for replication of the most important files (pictures and documents). This last part however is still WIP.
So far I've setup periodic SMART tests, Scrub Tasks and periodic Snapshot Tasks. I've also migrated most of my data at this point.

The Ubuntu VM got access to all cores and 48GB of Memory and Docker is running:

• Home Assitant
• Vaultwarden
• Portainer
• Cloudflare-DDNS
• Heimdall
• Nginx Proxy Manager
• Watchtower
• Obico
• Navidrome (which I still need to setup on the new server, this is currently still running on the old server)

Planned to add:

• Jellyfin (not sure if I'm ever gonna use it though, so far I've just mounted the NAS Folder via NFS in Kodi on my Shield TV and never had any issues, I'm not planning on streaming outside of my LAN)
• Sth. do manage Documents/PDF (if anyone got some recommendations please but them down below)
• Immich
• Tailscale (maybe?)
• ???

Currently the Ubuntu OS and docker containers are running of the boot drive, my goal is however, to have everything related to docker running of the Hexos SSD pool and to take periodic images of the VM and backup them to the HDD pool.

I've recently bought a Topton N100 which is now running OPNsense (and used as a Router) and I need to get PiHole working again, but I'll try to get it running on the N100. While changing my router I also changed my network range, meaning I'm no longer able to access Hexos until this bug is fixed. This is mostly fine, since I've been using it to manage access rights and I've got most things set up as I need them. (god damn, access managed is a true nightmare in truescale...)

I've an Unraid Server which I will use to replicate everything from Truenas and last but not least, my old NAS will be used as an off side Backup in case the building burns down. 🙃

That's all from my side, if you have any questions, remarks or recommendations please leave them below. Also if anyone knows where to buy good & cheap open server racks in europe please let me know.

Are you running all of these docker container apps natively in the Ubuntu VM? I'm just curious if your Ubuntu VM and HexOS VM are separate side-by-side systems or if you were able to get the TrueNAS version of the VM to play nice in Hex? I'm assuming since you're running Prox it's the former rather than the latter. If so, do you have a plan to easily migrate all those apps if/when the time comes? I suppose that means rather than the native apps coming to Hex in the future you'll be spinning up Docker or a layered VM when that becomes an option.. Also.. if it will be a layered VM how do you intend to combat the resource negotiations and network hurdles that often comes with nesting virtual environments?

 

Thanks.. sounds like you're off to a great start!

[PsychoWards]

2 hours ago, Zizuar said:

Are you running all of these docker container apps natively in the Ubuntu VM? I'm just curious if your Ubuntu VM and HexOS VM are separate side-by-side systems or if you were able to get the TrueNAS version of the VM to play nice in Hex? I'm assuming since you're running Prox it's the former rather than the latter. If so, do you have a plan to easily migrate all those apps if/when the time comes? I suppose that means rather than the native apps coming to Hex in the future you'll be spinning up Docker or a layered VM when that becomes an option.. Also.. if it will be a layered VM how do you intend to combat the resource negotiations and network hurdles that often comes with nesting virtual environments?

 

Thanks.. sounds like you're off to a great start!

Hey,

Yes they are all running natively in Ubuntu and Ubuntu is running alongside Hexos as a VM in proxmox. 🙂

I currently don't plan to switch to the Truenas docker and I suppose for most apps I'll likely still spin them up in the Ubuntu docker rather than using the Hexos Apps. This way I have more flexibility and can easily migrate them to another server if I want to in the future.

If I would use the Truenas VM feature in the future, I would ditch proxmox entirely and have Hexos running as the OS, because then, just for my use case I would have Proxmox running with just the Hexos VM and have the other VM(s) in Hexos? Might as well get rid of Proxmox and do it the propper way.

I might do this at some point in the future, depending on how those things are implemented but for the moment I'm fine with using Proxmox as the main OS.

[Zizuar]

On 12/27/2024 at 1:52 AM, PsychoWards said:

Hey,

Yes they are all running natively in Ubuntu and Ubuntu is running alongside Hexos as a VM in proxmox. 🙂

I currently don't plan to switch to the Truenas docker and I suppose for most apps I'll likely still spin them up in the Ubuntu docker rather than using the Hexos Apps. This way I have more flexibility and can easily migrate them to another server if I want to in the future.

If I would use the Truenas VM feature in the future, I would ditch proxmox entirely and have Hexos running as the OS, because then, just for my use case I would have Proxmox running with just the Hexos VM and have the other VM(s) in Hexos? Might as well get rid of Proxmox and do it the propper way.

I might do this at some point in the future, depending on how those things are implemented but for the moment I'm fine with using Proxmox as the main OS.

Out of curiosity then, what are you planning on using Hex for? Just simple network attached storage w/ raid failover availability?  It almost sounds redundant with what you have running on the side there.

[PsychoWards]

Exactly, Hexos is just a simple NAS with a ZFS Raid in my setup, where everything will be stored at and which is going to do backups to different drives/servers.

I will likely not use any of the fancy app or VMs features, I handle those things in VMs or opnsense directly.

[Zizuar]

5 hours ago, PsychoWards said:

Exactly, Hexos is just a simple NAS with a ZFS Raid in my setup, where everything will be stored at and which is going to do backups to different drives/servers.

I will likely not use any of the fancy app or VMs features, I handle those things in VMs or opnsense directly.

If I had the time right now I would probably give a proxmox setup a fun go. Other than some virtual windows servers at the office that used to house our on-prem CRM I've never really played with virtual machines. Think it would be a fun change.. Just have wayyy too many projects I'm behind on atm. 😞

[PsychoWards]

I've made some progress, Immich is installed and up and running and currently using my PCs GPU for the ML stuff... My current plan is to get my old 2080Ti installed and working in the server itself.

PaperlessNGX is now also running, which I use to store and archive all sorts of documents.

I've made the Hexos Gods very angry at me, by create a single drive pool with an external USB drive, which I use as a replication task target of some of my datasets.

Next on the list is getting rsync setup to my UNRAID and Synology NAS and at that point I think I should have enough backups of my most important files.

I'll post some pictures once I get to install the GPU.

[Zizuar]

3 different network storage machines? I hope at least one is totally off-site where you simply store backups. Data redundancy is important, but if they are all on the same network the entire set is vulnerable to malicious attacks. Only mention it because it sounds like a worst-case scenario you're planning for, and if so this needs to be a link in your chain.

[PsychoWards]

The Synology NAS is going to life at my parents house once the initial backup was made, so even if the entire building burns down, I'll have a backup. (So yes, indeed planning for the worst case)

I'm only backing up the important things, for the rest, snapshots need to be enough.