PCIe x16 NVMe Extension Card Compatibility

[ShinobiRen]

TL;DR: You cannot run an HBA and Asus Hyper M.2 x16 Gen 4 card and utilize all 4 NVMe drives (at least from the testing I have done - if anyone has been able to get this to work please let me know!)

 

I plan on posting my entire build process for my "stable" server once it is finally complete and I have all the resources together but wanted to post this in the off chance anyone has the same or similar setup I am running and runs into this issue. 

 

I am running a Ryzen 3900x CPU with an Asus ROG Strix X570-E motherboard. These are leftovers from a gaming computer upgrade I did earlier this year to get on AM5 but it got me thinking - Asus makes this handy NVMe expansion card here: https://www.asus.com/us/motherboards-components/motherboards/accessories/hyper-m-2-x16-gen-4-card/

I happened to have 4 spare NVMe drives from doing upgrades for my kids PCs last year that I haven't recycled yet so why not give it a shot?

I bought the expansion card, got it slotted, configured the BIOs to pass off all four to HexOS (TrueNAS) and was able to create a nice 3 TB pool in RaidZ1: 

 

No problem there. I went to add in an HBA (9300-8i) and the NVMe card dropped two drives from it. Tried every BIOs setting in the world I could think of but no dice - if I wanted to run the HBA and the NVMe expansion card I could only have two NVMe drives. 

 

Hope this helps if someone else has this same issue!

Edited 13 hours ago by ShinobiRen
Put the TL;DR at the top where it belongs

[PsychoWards]

8 hours ago, ShinobiRen said:

TL;DR: You cannot run an HBA and Asus Hyper M.2 x16 Gen 4 card and utilize all 4 NVMe drives

Hey, this is only true for your specific CPU & MoBo combi but not as a general statement.

According to your MoBo Manual, once you use 2 PCIe slots the bandwidth from the first slot drops from x16 to x8. And I suppose the ASUS card is using 4 PCIe lanes per SSD therefor with 8 lines you can only use 2 SSDs.

However if the first PCIe slot drops to x8 depends on the CPU generation and chipset, lots of the newer boards don't have this issue, Server Motherboards usually have more PCIe lanes then you will ever need and even some older platform don't have this problem neither.

So yes in your case it is unfortunately not possible to use the full 4 SSDs from the card in combination with a HBA card (due to HW limitations) , but for other setups this might not be a problem.