PsychoWards

That is very interesting, what's your intended use case if I may ask? I've spent some time thinking about L2ARC and SLOG/ZIL but I have never found any viable usecase for me, but I'm also new to those concepts.

Data will not be retained when creating a new pool. With 2 drives Hexos will only give you the option to have a mirrored setup, with 3+ drives Hexos will create a RaidZ1. There is currently no way to expand a mirrored to a RaidZ1 pool and I highly doubt that this will ever be possible. If you have a 2 drive mirrored pool and want to extend it, you need to backup all your data, create a new RaidZ1 pool (with all your drives) and if you want to add more drives in the future you can then do it.

Oh, that's awesome to hear. I've a N100 running Opnsense with only 8GB of memory, doubt I will ever need more, but it's really helpful information, I can imagine a lot of people are running N100 in there NAS. Would you mind sharing which module you got? 🙂

Can confirm, that's what I did and I have mirrored boot drives working flawlessly.

SSD caching in Truenas works probably very differently from what you expect them to. You have the possible to add the SSDs as L2ARC (ARC is responsible for keeping the frequently used files in memory and to predict and preload files which you are going to use. This is usually done in Memory, but you can expand it using a drive), this can improve read speeds, if the ARC isn't hitting the correct files but the L2ARC is, but only if you are using 10GBits network, or else the network will very likely be the bottleneck anyway. https://www.truenas.com/docs/references/l2arc/ Or you can add it as a SLOG which is used for synchronous writing. This is quite a complexe topic however. https://www.truenas.com/docs/references/slog/ If you are not using 10Gbit Networking and not using a lot of sycn. Writes I wouldn't bother and just either get a second boot drive to put as a mirror or make a "faster" mirrored pool with 2 SSDs. If you are using 10Gbit Networking, I still would not bother and go for a mirrored boot drive or mirrored fast SSD pool. Either way, L2ARC and SLOG/ZIL are the caching possibilities which you have.

Concerning question 1: Nowadays, the memory controller is baked into the CPU, so the CPU defines the supported Memory speed and quantity (mainly). In this case it is the 2666 which was mentioned and since everything higher then this is considered overclocking, and a lot of H370 boards don't support overclocking, that's why you are likely limited to this speed. Some Mid- to High End H370 boards support overclocking but I doubt that an OEM board does. Therefore, don't go with faster RAM and take more RAM instead.

Oh boy I was thinking about the 5600... Good catch with the PSU size! Thanks for correcting those things 🙂 You are right about the RAM it's not needed, but it's wasted potential, if you already go with relativ newish HW, you might want to consider using it to it's full potential. Also +1 for Intel CPU with QuickSync, that's the only reason why I went with Intel instead of AMD in my built.

Hey, The Ryzen 5 7600 does not have an integrated GPU, so make sure you have one ready when installing Hexos if you decided to use that CPU. If you only use 1 RAM stick you will only get half of the bandwidth of the memory, because it is running in single channel. If you want to have the full speed you need at least 2 modules so that they can run in dual channel. Since Hexos is doing a lot of caching in memory, you probably want to go dual channel to maximise performance. AM5 is using DDR5, which does include on-die ECC (Error Correction Code) and can correct a 1 bit error in the memory itself. It is, however, not the same kind of EEC which is used in server applications, which can detect bit flips outside of the memory as well. Powersupply: make sure to use a well known brand with gold or platinum rating, the Corsair RM650 for example can be had for around 90€, you don't need the 150€ which you have planned. Motherboard: you can always get an HBA card to add more SATA ports, however in that case you cannot add a dedicated GPU. But for most things, the integrated GPU should be more then enough. Another possibility would be a m.2 to SATA port adapter, which you plug in into an M.2 port, so there are possibilities on how to get more ports to your motherboard. M.2 SSDs: take something which doesn't have a lot of storage, since you cannot use it for anything else in Hexos, it will only run the OS.

@Ornival thank you so much for the very detailed explanations and remarks. This helps me a ton. Funny enough, my Server is already running in Proxmox anyway, so that part is already done =D I forgot to mention this earlier, but my Immich is running in Docker in an Ubuntu VM, it's not running in Hexos itself. I still have an 1060 with 3GB VRAM (which is really week both GPU an VRAM) and an 2080Ti, which however crashes while gaming. But maybe only the rasterization part is causing issues, the CUDA part may be fine, I'll have to test this. If the 2080Ti isn't usable, I will go on the hunt for one of your recommended GPUs if I find a good deal. I don't think that a lot of VRAM will be required for my use cases, because must programms with machine learning capabilities for HomeLabs are meant to run on any kind of HW and to not specifically need a high end GPU with a lot of VRAM. Another example would be paperless ngx, which also uses machine learning but doesn't even support HW acceleration. So I technically don't require a GPU, but the thinkering is the fun part.

I believe he meant as a boot drive not as the storage drives 🙂 If so, a HDD as a boot drive shouldn't be an issue, at least not with Truenas itself, but I don't believe that Hexos adds much overhead which would require an SSD to work properly. The NVME/M.2 SSD is often recommended because it frees up 1 (or in case of a mirror 2) SATA port, which there are not many left on modern consumer grade HW and because they aren't that expensive anymore and consum less energy. The boot might take longer, but most of the OS runs in Memory anyway. So if you are fine with longer boot time, decent HDDs should be fine.

Don't buy Synology, you can only use their own DSM OS but not Hexos. Make sure it has an x86 CPU (Intel or AMD) and not an ARM CPU (Hexos doesn't run on ARM) and that it has a video output.

Normally, the boot drive dying isn't an issue (except for the fact that you have to buy a new drive) as long as you have a backup off your settings. You would just reinstall the OS, import the backup and you are good to go. However since Hexos is still in Beta, this is currently not possible, there is no option to skip the pool creation during setup. This will come in the future but it is not there yet. Until then, if your boot drive stops working, your are going to install truenas directly and import the pools (the pool information is stored on the drives, which means you can directly read it from them) this way you can still access your data, however you have lost the Hexos part and can only switch back to Hexos once it supports pool import. Since the data drives using ZFS you have to read them as such, you cannot just mount them individually and try to get data from them. But as I said a boot drive dying will not result in data loss, you would only need to switch to truenas for the time being.

Hey, thank you so much for this detailed answer. My current setup consists of an i5 14600K and no dedicated GPU in the server and Immich is connecting to my PC to have the ML stuff run on my 3080Ti. The "problem" however is that my PC is not running 24/7 and that I don't want Immich to hit my GPU while i'm gaming. Therefore I thought maybe a dedicated GPU would be helpful for this part. I don't want to upgrade my GPU, especially since it has a water cooling block installed, and I don't want to put a customer water cooling loop in my server. My only current usecase is indeed Immich, (which does not support ML on the iGPU) but in the feature some AI recognition for security cameras might be added as well. I'm not looking to get a high end GPU, but maybe something lower class or used Nvidia P model GPUs (P40) or sth. Similar in the 300€/$ price range, since they have way faster single and double FP performance then any high end 4000/5000 series GPU. For transcoding I'm using my iGPU anyway.

It depends on your use case and what you need. Do you mainly need the NAS and the VMs are just "nice to have" or are the VMs important and Hexos is just one part/VM of many for you? If I see that you want to spin up Ubuntu and Windows, you might as well go with Proxmox and Hexos as a VM, however this comes with it's own challenges. A lot of guides on how to install truenas are outdated/recommend suboptimal settings and you can only pass-through your GPU to 1 VM, meaning if you pass through your GPU to hexos for plex streaming, you can not use it in your ubuntu VM for machine learning in Frigate NVR (this is just an example, not sure Frigate NVR uses GPU acccleration). But on the plus sidey Proxmox is specifically developed to manage VMs and you typically find a lotnof guides and help for itm Whereas if you use barebone Hexos, you don't have to fiddle with the VM pass-through stuff for Hexos, but the VM managing part might be less optimal and less performant. According to the Intel spec page, your CPU supports virtualisation. However, it will likely be on the week side if you want to run those 3 OSes together. Also Hexos requires minimum 8GB of RAM, meaning you need to significantly upgrade your memory. Your CPU supports max up to 32GB of RAM and that is probably what you should aim for, since 16GB might be to little.

Exactly, Hexos is just a simple NAS with a ZFS Raid in my setup, where everything will be stored at and which is going to do backups to different drives/servers. I will likely not use any of the fancy app or VMs features, I handle those things in VMs or opnsense directly.

For Mac you can follow this excellent guide: For Windows what kind of backups do you want to do? Would it be enough to just store the important data on the NAS directly?

Has anyone some experience or tips and recommendation which GPU or similar to get for HW acceleration and/or machine learning? Immich for example has some machine learning stuff, which I've currently offloaded to my PC, however I'm looking to get a dedicated GPU which I can directly plug in the server. If anyone has some recommendations on what to get and some tips I would greatly appreciate it. I'm not planning on running my full blown ChatGPT replacing or sth. similar, just to use the machine learning in Immich and to be ready for future lightweight applications which might need some raw power.

Hey, Looks like a proper built in my eyes, with a lot of potential to grow as you intend to do. Just some remarks; currently RaidZ2 is not supported by Hexos, Hexos will always create a RaidZ1. You can manually create a RaidZ2 pool in Truenas but this might have unforeseen impacts and consequences. You might want to consider buying a second boot drive and having the boot drives running as a mirror. This can save you a lot of headaches in case the boot drive fails, because right now, it's not possible to import pools into Hexos, meaning if the drive dies you cannot reinstall Hexos, because it will wipe your drives and even if pool import is supported in the future, it might still be worth the consideration, especially since you are already going full server build anyway.

I'm not sure that you can easily install anything on a Synology NAS, they are pretty locked down and would require a lot of tinkering to even install Hexos. So I would not go down that route. Also make sure that you get something with a video output, or else installation and management in case you locked yourself out will be a nightmare. Also it is not possible to use the boot drive in a Pool, meaning if you have a 3 storage bays, 1 drive will be used as a boot driving, leaving you with only 2 drives left. Take this into consideration, that you also need space for a dedicated boot drive.

Hey, Yes they are all running natively in Ubuntu and Ubuntu is running alongside Hexos as a VM in proxmox. 🙂 I currently don't plan to switch to the Truenas docker and I suppose for most apps I'll likely still spin them up in the Ubuntu docker rather than using the Hexos Apps. This way I have more flexibility and can easily migrate them to another server if I want to in the future. If I would use the Truenas VM feature in the future, I would ditch proxmox entirely and have Hexos running as the OS, because then, just for my use case I would have Proxmox running with just the Hexos VM and have the other VM(s) in Hexos? Might as well get rid of Proxmox and do it the propper way. I might do this at some point in the future, depending on how those things are implemented but for the moment I'm fine with using Proxmox as the main OS.

I was in the midst of putting together my new server, which was supposed to run TrueNas, when I received the e-mail that Hexos was available and that's how I ended up here. 🙂 This server was supposed to replace my 10 year old NAS and to replace my current Windows Server running docker and some game servers when needed. The HW specs are: CPU: Intel i5 14600K with Thermal Grizzly PhaseSheet PTM (Phase Shifting Pad will not try up compared to thermal paste, thus will not require maintenance in the future) CPU-Cooler: Noctua NH-D9L Memory: G.Skill Ripjaws S5 128GB DDR5-5600 (4x32GB) MoBo: ASRock Z790 PRO RS PSU: Corsair RM650 NIC: QSFPTEK 10G SFP+ (2x Intel X520-DA2) Boot Drives: 2x WD Red SN700 500GB HDDs: 6x Seagate IronWolf 4TB SSDs: 2x Samsung 990 Evo 1TB SSD cooler: 3x be quiet! MC1 Pro (since the MoBo only has 1 m.2 slot with a cooler) external HDD: 1x Seagate Desktop Drive 8TB case: Inter-Tech IPC 4U-4129L UPS: APC BX1600MI I didn't want to get 10 year old used server grade HW, I wanted something newer, faster and more energy efficient, thus the HW above. Yes I know that it would have been better to gett ECC Memory, but ECC Memory and new consumer grade HW is an absolute nightmare, but at least DDR5 got some On-Die EEC which is better then nothing. Neither my old Synology NAS, nor my Unraid Server are rocking ECC memory, so i'm taking my chances. The server is running Proxmox with one VM running Hexos and another running Ubuntu Server and it's configured to shutdown once the UPS is below 30% of battery power. Both NICs are configured as a Bond in active backup mode, meaning if 1 fails, the other will automagically take over without any interrupt of data transfer (I did test this, it's working great 🙂 ). Both of the VMs are setup with UEFi instead of BIOS and as Q35 Machines instead of the default i440fx. (because PCI-e passthrough requires Q35 and I just wanted to be safe) I'm planning on setting up a 3. VM running Windows in case I need it for some game server or if I want to quickly deploy something (my windows skill are far superior compared to my linux skills) The Hexos VM has 6 Cores and 64GB of Memory, 1 RaidZ2 Pool with the 6 HDDs, 1 Mirrored Pool with the SSDs and the external HDD will be used for replication of the most important files (pictures and documents). This last part however is still WIP. So far I've setup periodic SMART tests, Scrub Tasks and periodic Snapshot Tasks. I've also migrated most of my data at this point. The Ubuntu VM got access to all cores and 48GB of Memory and Docker is running: Home Assitant Vaultwarden Portainer Cloudflare-DDNS Heimdall Nginx Proxy Manager Watchtower Obico Navidrome (which I still need to setup on the new server, this is currently still running on the old server) Planned to add: Jellyfin (not sure if I'm ever gonna use it though, so far I've just mounted the NAS Folder via NFS in Kodi on my Shield TV and never had any issues, I'm not planning on streaming outside of my LAN) Sth. do manage Documents/PDF (if anyone got some recommendations please but them down below) Immich Tailscale (maybe?) ??? Currently the Ubuntu OS and docker containers are running of the boot drive, my goal is however, to have everything related to docker running of the Hexos SSD pool and to take periodic images of the VM and backup them to the HDD pool. I've recently bought a Topton N100 which is now running OPNsense (and used as a Router) and I need to get PiHole working again, but I'll try to get it running on the N100. While changing my router I also changed my network range, meaning I'm no longer able to access Hexos until this bug is fixed. This is mostly fine, since I've been using it to manage access rights and I've got most things set up as I need them. (god damn, access managed is a true nightmare in truescale...) I've an Unraid Server which I will use to replicate everything from Truenas and last but not least, my old NAS will be used as an off side Backup in case the building burns down. 🙃 That's all from my side, if you have any questions, remarks or recommendations please leave them below. Also if anyone knows where to buy good & cheap open server racks in europe please let me know.

This and if you server is not reachable (maintenance, updates, ISP problem, ISP forced disconnecteds and reconnects, power outage, a wrong manipulation on the server, house keeper unplugging the server to plug in the vacuum cleaner etc...), you are not going to get any mails and this becomes problematic with missed invoices and other important mails. Therefore while it certainly is a fun and interesting topic, I would only use it for emails which you don't mind not receiving, but I would not use it for important things, but that's up for everybody to decide themselves.

You need to login into Truenas, go to storage -> Manage Devices -> select your pool and then the first drive. On the right side on the Disk Info part press "Edit" and there you have the possibility to set the HDD Standby and/or Advanced Power Management. Under Advanced Power Management everything from Level 1 to 127 will spin down the drive, from 128 to 254 will not spin down the drive. You have to do this for every HDD. I don't know how Hexos is reacting to this and which side effect it can have, also for other people reading this, spinning down drives is generally not recommended because the spin ups are putting wear on the drives, and drives are vulnerable during spin ups. When drives are spinning there is an air cushion which is keeping the head of the disks, while this is not the case during a spin up and a head crash can occur. Also when leaving a drive sitting without spinning it can have a negative impact on the lubrication of the bearing. Also make sure that the drive doesn't not cool down to much, this is especially import in the winter if the server is sitting in a cellar or basement where the air temperature can drop and this can also cool down your drive significantly, depending on your setup. There have been plenty of perfectly working drives without issues, which did not recover from a spin up. So be warned, that it might shorten the lifespan of a drive. Also you might want to look into enabling the different C-States in the Bios, this can also save a significant amount of energy.

Hey, Concerning the RAM, go with ECC modules. Just a quick heads-up, Hexos is using ZFS, which is different then raid and has a different naming scheme. For example the RAID1 in ZFS is called mirrored, RaidZ1 in ZFS would be similar to Raid5, RaidZ2 would be similar to Raid6. Therefor to not get confused between Raid1 (ZFS mirrored) and ZFS RaidZ1 (Raid5) I will use the ZFS terms. Mirrored boot drive is the way to go with the current version of Hexos. Since pool import is not supported at the moment, if you only have 1 boot drive and if your boot drive bites the dust, there is currently no way to use hexos again without loosing all your data. You would need to install truenas and use truenas until Hexos supports pool import. I am using WD Red Plus and Seagate Ironwolf drives, they are meant for NAS and should last longer in theory. However a lot of people are using desktop grade drives without much issue. My only recommendation would be to make sure you get CMR drives and not SMR drives (meaning don't buy for example normal WD Red drives, either use the Red Plus or Red Pro for those) If you are only using it as a NAS and don't want to host a lot of other things and VMs, 1 CPU would be more than enough, this will save on energy. Additionally, you can also activate some of the C-States in the BIOS to save even more energy on idling and you can disable not used components (audio chip on consumer grad HW, onboard NICs if an external NIC is used etc...) Stopping drives from spinning would also save some energy, however this is generally not recommended since it increases the wear on the drives, spinning drives up is one of the more frequent cases where a drive can die.

No, you cannot change from a RaidZ1 to a RaidZ2, this is defined once during the creation of the pool, but changing it requires to destroy and recreate the pool with RaidZ2, which will wipe all the data from the disks. If you want to have a RaidZ2 you either need to wait until Hexos supports it or create it manually in truenas and live with possible side effects in Hexos or ditch Hexos for the time being and switch to a truenas installation and migrate back to Hexos once pool migration because available some time after version 1.0.