Terms I suggest you understand before reading this: Network Interface Card (NIC), PCIE, Ethernet
I use my main pc for video editing. I like most people only have a 1 gbit port on my pc and nas. All my local networking gear (switches, routers) also only support 1 gbit. This wasn't cutting it for me when transferring video files back and forth from my nas. The 1 gbit connection for the rest of my house was fine as this is plenty fast for the rest of my devices as their most intensive use would be 4K HDR Plex streams or web browsing.
Just like Ethernet is the protocol most popular for wired network connections today there is a technology called Infiniband that is a networking protocol heavily used in enterprise gear. Mellanox (now owned by Nvidia) has a product line called ConnectX. It's what they call their Infiniband gear. Currently ConnectX-8 is their latest iteration that supports 800 gbit. Since this is the latest tech these are still very expensive. The older editions get dumped on ebay in bulk when corporations upgrade their systems. The one I use is ConnectX-3 and supports up to 56gbit but the two cards I purchased support up to 40gbit. These cards are PCIE 3.0 and use an X8 slot. They will fit in an X4 slot but will be limited to 32gbit as each pcie 3.0 lane supports 8gbps.
The specific cards I run are the dual port Mellanox MCX354-QCBT.
Normally with fiber optic cables you would need a transceiver. These are what plug into the ports on fiber optic networking gear to convert the signals into light. The reason these are not just built into networking gear is because they come in different strengths depending on cable length. If you use a transceiver made for distances over 14 miles on a 100ft cable you'll burn out the transceiver on the other end.
These transceivers come in different form factors to fit different ports. The ones in these Mellanox cards I linked above use a QSFP+ port. They are not interchangeable. SFP with SFP, SFP+ with SFP+, QSFP+ with QSFP+, QSFP28 with QSFP28.
Now my NAS and my PC are in the same room so I can simplify my solution by using a direct attach cable (dac). This is a cable that already has transceivers on the ends and will plug directly into my connectx-3 cards. The DACs use a copper wire instead of fiber optics so they are limited to about 20 feet. Any longer than that and you will need to use transceivers and a fiber optic cable.
This is the DAC I used but you can find cheaper options in varying lengths.
Once I get my new drives to build my HexOS nas I will write an actual how to guide on how to set everything up in Windows and HexOS as you need to manually set the IP address for these two cards to be able to talk to each other over the dac.
I have had this setup for a little over a year now so you may be able to find faster nics such as Connectx-4 cards
Additional info for more advance users. On some of these ConnectX cards you can change the network protocol on the ports in the card. On my specific cards I can use them in Infiniband or Ethernet. They support 10gbe so with a qsfp+ to rj45 transceiver or media converter you can connect them to your 10gb ethernet switch.