PsychoWards

For Mac you can follow this excellent guide: For Windows what kind of backups do you want to do? Would it be enough to just store the important data on the NAS directly?

Has anyone some experience or tips and recommendation which GPU or similar to get for HW acceleration and/or machine learning? Immich for example has some machine learning stuff, which I've currently offloaded to my PC, however I'm looking to get a dedicated GPU which I can directly plug in the server. If anyone has some recommendations on what to get and some tips I would greatly appreciate it. I'm not planning on running my full blown ChatGPT replacing or sth. similar, just to use the machine learning in Immich and to be ready for future lightweight applications which might need some raw power.

Hey, Looks like a proper built in my eyes, with a lot of potential to grow as you intend to do. Just some remarks; currently RaidZ2 is not supported by Hexos, Hexos will always create a RaidZ1. You can manually create a RaidZ2 pool in Truenas but this might have unforeseen impacts and consequences. You might want to consider buying a second boot drive and having the boot drives running as a mirror. This can save you a lot of headaches in case the boot drive fails, because right now, it's not possible to import pools into Hexos, meaning if the drive dies you cannot reinstall Hexos, because it will wipe your drives and even if pool import is supported in the future, it might still be worth the consideration, especially since you are already going full server build anyway.

I'm not sure that you can easily install anything on a Synology NAS, they are pretty locked down and would require a lot of tinkering to even install Hexos. So I would not go down that route. Also make sure that you get something with a video output, or else installation and management in case you locked yourself out will be a nightmare. Also it is not possible to use the boot drive in a Pool, meaning if you have a 3 storage bays, 1 drive will be used as a boot driving, leaving you with only 2 drives left. Take this into consideration, that you also need space for a dedicated boot drive.

Hey, Yes they are all running natively in Ubuntu and Ubuntu is running alongside Hexos as a VM in proxmox. 🙂 I currently don't plan to switch to the Truenas docker and I suppose for most apps I'll likely still spin them up in the Ubuntu docker rather than using the Hexos Apps. This way I have more flexibility and can easily migrate them to another server if I want to in the future. If I would use the Truenas VM feature in the future, I would ditch proxmox entirely and have Hexos running as the OS, because then, just for my use case I would have Proxmox running with just the Hexos VM and have the other VM(s) in Hexos? Might as well get rid of Proxmox and do it the propper way. I might do this at some point in the future, depending on how those things are implemented but for the moment I'm fine with using Proxmox as the main OS.

I was in the midst of putting together my new server, which was supposed to run TrueNas, when I received the e-mail that Hexos was available and that's how I ended up here. 🙂 This server was supposed to replace my 10 year old NAS and to replace my current Windows Server running docker and some game servers when needed. The HW specs are: CPU: Intel i5 14600K with Thermal Grizzly PhaseSheet PTM (Phase Shifting Pad will not try up compared to thermal paste, thus will not require maintenance in the future) CPU-Cooler: Noctua NH-D9L Memory: G.Skill Ripjaws S5 128GB DDR5-5600 (4x32GB) MoBo: ASRock Z790 PRO RS PSU: Corsair RM650 NIC: QSFPTEK 10G SFP+ (2x Intel X520-DA2) Boot Drives: 2x WD Red SN700 500GB HDDs: 6x Seagate IronWolf 4TB SSDs: 2x Samsung 990 Evo 1TB SSD cooler: 3x be quiet! MC1 Pro (since the MoBo only has 1 m.2 slot with a cooler) external HDD: 1x Seagate Desktop Drive 8TB case: Inter-Tech IPC 4U-4129L UPS: APC BX1600MI I didn't want to get 10 year old used server grade HW, I wanted something newer, faster and more energy efficient, thus the HW above. Yes I know that it would have been better to gett ECC Memory, but ECC Memory and new consumer grade HW is an absolute nightmare, but at least DDR5 got some On-Die EEC which is better then nothing. Neither my old Synology NAS, nor my Unraid Server are rocking ECC memory, so i'm taking my chances. The server is running Proxmox with one VM running Hexos and another running Ubuntu Server and it's configured to shutdown once the UPS is below 30% of battery power. Both NICs are configured as a Bond in active backup mode, meaning if 1 fails, the other will automagically take over without any interrupt of data transfer (I did test this, it's working great 🙂 ). Both of the VMs are setup with UEFi instead of BIOS and as Q35 Machines instead of the default i440fx. (because PCI-e passthrough requires Q35 and I just wanted to be safe) I'm planning on setting up a 3. VM running Windows in case I need it for some game server or if I want to quickly deploy something (my windows skill are far superior compared to my linux skills) The Hexos VM has 6 Cores and 64GB of Memory, 1 RaidZ2 Pool with the 6 HDDs, 1 Mirrored Pool with the SSDs and the external HDD will be used for replication of the most important files (pictures and documents). This last part however is still WIP. So far I've setup periodic SMART tests, Scrub Tasks and periodic Snapshot Tasks. I've also migrated most of my data at this point. The Ubuntu VM got access to all cores and 48GB of Memory and Docker is running: Home Assitant Vaultwarden Portainer Cloudflare-DDNS Heimdall Nginx Proxy Manager Watchtower Obico Navidrome (which I still need to setup on the new server, this is currently still running on the old server) Planned to add: Jellyfin (not sure if I'm ever gonna use it though, so far I've just mounted the NAS Folder via NFS in Kodi on my Shield TV and never had any issues, I'm not planning on streaming outside of my LAN) Sth. do manage Documents/PDF (if anyone got some recommendations please but them down below) Immich Tailscale (maybe?) ??? Currently the Ubuntu OS and docker containers are running of the boot drive, my goal is however, to have everything related to docker running of the Hexos SSD pool and to take periodic images of the VM and backup them to the HDD pool. I've recently bought a Topton N100 which is now running OPNsense (and used as a Router) and I need to get PiHole working again, but I'll try to get it running on the N100. While changing my router I also changed my network range, meaning I'm no longer able to access Hexos until this bug is fixed. This is mostly fine, since I've been using it to manage access rights and I've got most things set up as I need them. (god damn, access managed is a true nightmare in truescale...) I've an Unraid Server which I will use to replicate everything from Truenas and last but not least, my old NAS will be used as an off side Backup in case the building burns down. 🙃 That's all from my side, if you have any questions, remarks or recommendations please leave them below. Also if anyone knows where to buy good & cheap open server racks in europe please let me know.

This and if you server is not reachable (maintenance, updates, ISP problem, ISP forced disconnecteds and reconnects, power outage, a wrong manipulation on the server, house keeper unplugging the server to plug in the vacuum cleaner etc...), you are not going to get any mails and this becomes problematic with missed invoices and other important mails. Therefore while it certainly is a fun and interesting topic, I would only use it for emails which you don't mind not receiving, but I would not use it for important things, but that's up for everybody to decide themselves.

You need to login into Truenas, go to storage -> Manage Devices -> select your pool and then the first drive. On the right side on the Disk Info part press "Edit" and there you have the possibility to set the HDD Standby and/or Advanced Power Management. Under Advanced Power Management everything from Level 1 to 127 will spin down the drive, from 128 to 254 will not spin down the drive. You have to do this for every HDD. I don't know how Hexos is reacting to this and which side effect it can have, also for other people reading this, spinning down drives is generally not recommended because the spin ups are putting wear on the drives, and drives are vulnerable during spin ups. When drives are spinning there is an air cushion which is keeping the head of the disks, while this is not the case during a spin up and a head crash can occur. Also when leaving a drive sitting without spinning it can have a negative impact on the lubrication of the bearing. Also make sure that the drive doesn't not cool down to much, this is especially import in the winter if the server is sitting in a cellar or basement where the air temperature can drop and this can also cool down your drive significantly, depending on your setup. There have been plenty of perfectly working drives without issues, which did not recover from a spin up. So be warned, that it might shorten the lifespan of a drive. Also you might want to look into enabling the different C-States in the Bios, this can also save a significant amount of energy.

Hey, Concerning the RAM, go with ECC modules. Just a quick heads-up, Hexos is using ZFS, which is different then raid and has a different naming scheme. For example the RAID1 in ZFS is called mirrored, RaidZ1 in ZFS would be similar to Raid5, RaidZ2 would be similar to Raid6. Therefor to not get confused between Raid1 (ZFS mirrored) and ZFS RaidZ1 (Raid5) I will use the ZFS terms. Mirrored boot drive is the way to go with the current version of Hexos. Since pool import is not supported at the moment, if you only have 1 boot drive and if your boot drive bites the dust, there is currently no way to use hexos again without loosing all your data. You would need to install truenas and use truenas until Hexos supports pool import. I am using WD Red Plus and Seagate Ironwolf drives, they are meant for NAS and should last longer in theory. However a lot of people are using desktop grade drives without much issue. My only recommendation would be to make sure you get CMR drives and not SMR drives (meaning don't buy for example normal WD Red drives, either use the Red Plus or Red Pro for those) If you are only using it as a NAS and don't want to host a lot of other things and VMs, 1 CPU would be more than enough, this will save on energy. Additionally, you can also activate some of the C-States in the BIOS to save even more energy on idling and you can disable not used components (audio chip on consumer grad HW, onboard NICs if an external NIC is used etc...) Stopping drives from spinning would also save some energy, however this is generally not recommended since it increases the wear on the drives, spinning drives up is one of the more frequent cases where a drive can die.

No, you cannot change from a RaidZ1 to a RaidZ2, this is defined once during the creation of the pool, but changing it requires to destroy and recreate the pool with RaidZ2, which will wipe all the data from the disks. If you want to have a RaidZ2 you either need to wait until Hexos supports it or create it manually in truenas and live with possible side effects in Hexos or ditch Hexos for the time being and switch to a truenas installation and migrate back to Hexos once pool migration because available some time after version 1.0.

At least for Truenas, you can backup your config and store it wherever you like. In the case of a boot drive failure, you would just reinstall truenas, reimport the config and you are good to go, no need to do an image of the boot drive itself. This will not work with hexos, since hexos will create new pools and wipe the drives. But using Truenas instead would be an option to recover the data from the drives until importing pools is possible in hexos.

I cannot help you with the Immich migration, however currently it's not possible to create a pool in Hexos with 1 single drive, you need at least 2 drives so that Hexos can create at least a mirror. If, however you want to expand your pool in the future you need at least 3 drives during the creation. Those are the current limitations of hexos, this will however change in the future. So your current plan of sacrificing 1 ssd for the pool creation will not work.

Are those WD Red or Red Plus or Red Pro? Simple WD Red are not ideal for a NAS, despite WD marketing them as NAS drives, since they a SMR an not CMR drives, the performance can suffer. Other than that, if you can spare the extra capacity lost go with a RaidZ2, meaning you will have 8TB storage and you can lose 2 drives without losing any data. However, at the moment Hexos will always create a RaidZ1 meaning you can only lose 1 drive but have 10TB storage.

And Vaultwarden if that's possible

How do you want to do this? There is no reliable way to measure the consumption of most of the parts, some parts are not even read be the motherboard (HDD consumption, PSU losses etc.) Your best bet is going to buy a cheap socket power meter, this does it way more reliable, you know how much is actually pulled from the socket and depending on the model you can connect it to Home Assistant for example.

@The Callorhinchus I'm not sure if you got an invite already but @Adster mentioned in one of the categories, which you see after you got the invitation, that the everything is running fine when installing Hexos as a VM in Promox, so at least there's that. I guess most people using proxmox just use truenas directly, the Hexos target audience probably doesn't want or need the increased complexity/possibilities? I'll use proxmox since Hexos currently doesn't support all the applications which I need and I will spin up dockers in a VM in proxmox.

I'm interested in this as well, is it better to use Proxmox for the VM stuff and use Hexos as a NAS only or should we just use Hexos for both the storage and VMs?

Yes please, this should be done rather sooner than later.