The HULK... my first HexOS NAS...

[jamaican.mekrazy]

All righty then...

My first-ever NAS was using TrueNAS and was a trial-by-fire, to say the least.  This time, after watching a HexOS demo by Linus (LTT) I decided to give it a try.  It went super smoothly and I'm very impressed.  My hardware configuration:

 

#  PlinkUSA RackBuy IPC-G3550X 3U Server Chassis   

#  MSI MAG B550M Mortar MAX WiFi Motherboard  

#  AMD Ryzen 5 5600G 4.4GHz 6-Core 12-Thread Processor  

#  32GB (2x16GB) G.SKILL Trident Z-Series DDR4 3200MT/s CL14-14-14-34 Memory

#  1TB Kingston FURY Renegade PCIe Gen 4.0 NVMe OS Drive

#  (12) 8TB SAMSUNG 870 QVO SATA III SSDs

#  10Gb Dual RJ45 (Intel X540-T2) PCIe 3.0 x8 Network Card

#  LSI Logic 9300-16i 16Port 12Gb/s SAS PCIe 3.0 x 8 HBA Controller

#  (3) ICY DOCK 6-Bay 2.5” SATA HDD / SSD Hot Swap Tool-Less Backplane Enclosure with Dual Cooling Fan for 5.25” Bay

#  FSP Twins Pro ATX PS2 1+1 Dual Module 500W Certified Efficiency ≥90% Hot-swappable Redundant Digital Power Supply

So without further ado I bring you The HULK...  

I'm currently in the process of transferring about 42TB of media from a Yottamaster 5-Bay USB 3.2 HDD enclosure to the new pool.  Currently I'm seeing an average transfer rate of about 30MB/s which is super slow considering everything is 10G (switch, nics) and using Cat8 cabling.  I have checked the link speeds and they are all reporting 10G.  It might be a device cache limitation.  The first ~2GB of any transfer is faster (>1GB/s).  I think I may have to look into a second M.2 for a cache drive.  That's my opinion, I welcome yours.  :)

[DomSmith]

Looks like an interesting build.

If you're having speed issues you could try a large single file transfer with something like  Blu-ray rip and try to see where it's going wrong.

I've personal had the following issues:

1 - Bad drivers (Microsoft auto-installed) for the NIC on the windows machine I was transferring from.

2 - Insufficient PCIe bandwidth to the NIC and HBA on the NAS. Use the lspci command from the Linux shell to check for this.

3 - HBA and NIC overheating as they were designed to be aggressively cooled by server chassis fans.

4 - 'cache-less' SSDs that slow down (a lot) under extended write loads.

 

I also found Crystal DiskMark quite useful for this as it isn't dependant on a source disk. You can look at the results and ponder how they might corrospond to things like PCIe link rates etc.

[PsychoWards]

That does look like a proper setup =D

Is everything connected to your UPS?

Concerning your copying speed, can you try copying from and to another device, to see which system might be the issue? If you're not finding a conclusive answer, you can create another topic in the support forum to get some more help on this matter.

 

1 hour ago, jamaican.mekrazy said:

 I think I may have to look into a second M.2 for a cache drive.

There is no write cache in Truenas/Hexos there is only L2ARC. This is an extension of the ARC which is running in memory. The ARC is buffering files which truenas thinks you might use next, to have them ready in memory and not needing to ready it from the drives. The L2ARC is an extension and it's keeping more files on the cache drive. It is a read only cache.

[jamaican.mekrazy]

@DomSmith Thank you for the ideas, Dom.  I'll try them.  Yes, the NAS is connected to it's own 1000VA UPS (the smaller one) via the lower power distribution bar.  The remainder of my network (main PC, RUST game server, NUC 8i7 torrent PC and a MinisForum SFF Linux PC) are connected to the 1500VA UPS via the upper bar.  Everything is handled by an auto-engaging propane generator that runs the whole house.  30 seconds after power failure and I'm back up and off UPS power.  I removed the 24-port patch bar sometime after taking the photo, as I didn't really need it.

@PsychoWards Thank you for the thumbs-up!  So the L2ARC cache is for reads only?

Update: Since the initial post the transfer speed miraculously bumped itself up to 235MB/s and has remained steady for hours.  I'm not going to complain, though I do wonder... Is my connection between PC and NAS travelling from PC to Switch to NAS (as I presume) or from PC to Switch to Router to Switch to NAS?  If the later, it would explain the 235MB/s limit as the router only supports 2.5GB/s.  My apology in advance if this question needs to be moved to a new forum category.

Edited January 27 by jamaican.mekrazy

[PsychoWards]

6 minutes ago, jamaican.mekrazy said:

So the L2ARC cache is for reads only?

Exactly and depending on your usecase it might be hit or miss, if the ARC is caching the wrong files. For most user's it's not really useful except if you absolutely need the faster read speed of an m.2 SSD or your are doing a lot of video/image editing. If you just use your NAS as a backup, media storage solution, the L2ARC is very likely not needed.

It's possible that you were transfering a lot of small files earlier, which could explain the 30MB/s you saw earlier.

The connection is always taking the shortest path, if your PC and Server are connected to the same switch, they will not pass through the router.

[jamaican.mekrazy]

The HULK is just a media storage/streaming device.  Once my files are transferred I'll be able to get a better picture of the read speeds during streaming (at most, two simultaneous).  It may be that L2ARC might help me there.

As to the files, they are all standard 1.2GB-1.8GB 1080p MP4 and MKV files (though I do have an occasional 50GB+ file).  I do realize that a mish-mash of file sizes can have throughput results like I was seeing but I'm almost certain that is not the case here.  It is possible that cached data took a while to transfer from SLC to permanent storage, and once it caught up things perked up?  I also found that the drives I'm using are notorious for slow write speeds due to said cache type.  I also want to look into something called overprovisioning which allows the SLC cache size to be increased... perhaps a performance gain is to be had there, but I'll only be able to test this out if I can find some Linux software similar in functionality to Samsung's Magician.

I know... I talk a lot.  It's not often I get to pick the brains of a community of high-thinking people.  I very much appreciate all your help.

[Dylan]

5 hours ago, jamaican.mekrazy said:

Everything is handled by an auto-engaging propane generator that runs the whole house.  30 seconds after power failure and I'm back up and off UPS power.

My mans got a back up propane genny???? Link pease!!

[ubergeek]

3 minutes ago, Dylan said:

My mans got a back up propane genny???? Link pease!!

 

Generac ?

[jamaican.mekrazy]

@ubergeek I wish.  It's a Briggs & Stratton 12kW (01938-0) propane standby generator that my folks had installed when our 2-story Cape was built back in '89? I think.  It gets yearly servicing and is still going strong.

[jamaican.mekrazy]

@Dylan  https://www.briggsandstratton.com/na/en_us/support/results.multiStepManualSearch.html?searchrequested=01938-0

 

[nugglet]

Very impressive setup. Do you have all the 8TB SSDs in a single pool?

[jamaican.mekrazy]

@nugglet TGIF!  Yes.  I began with the default pool size that HexOS recommended.  Then before transferring data, I expanded it one SSD at a time until the maximum recommended drives had been reached.  All 37TB of data has been transferred.  To date, 100% healthy and zero errors.

[Joshua Brown]

The build itself looks similar to how I have mine as well except no fans (which makes sense since your running those sweet sweet SSDs)! Also, always love to see a server rack, just completes a build imo

[jamaican.mekrazy]

Thanks, Joshua!