I'm not knowledgeable about Plex on HexOS but using Intel Quicksync for hardware transcoding on a maybe 8th gen or newer Intel CPU which is easily the cheapest way to do this from what I know. Now does Plex on HexOS support Intel Quicksync hardware transcoding?
Here are example NAS builds with used hardware.
https://forums.serverbuilds.net/t/guide-nas-killer-6-0-ddr4-is-finally-cheap/13956
Also check out the earlier NAS Killer 4.0, 4.1, 5.0 based on older hardware