Error too many open files even after updates

[Dirtscootdesigns5956]

Alert from DuVall-Backup:

Failed to check for alert SyslogNg: [Errno 24] Too many open files

 

 This is the error I keep getting sent to my email, never had an issue before during about a year of hex os use. Any ideas on how to fix it? I have recently made sure everything is up to date! Thanks! Loving the product so far! So worth the lifetime licence!

[mill3000]

Thanks for posting what you found. I informed a team member to review this. If everything is in order, they will submit a ticket to IX (TrueNAS) for the next system update. We appreciate all the members helping out to make HexOS better for everyone. One of the postives of the internet and forums that don't get appriciated enough.

[mill3000]

Hello,

One way that could fix this is to do the following. 

Update to TrueNAS version 25.10.2.1. which applies permanent resource limit fixes.  

Thank you

[Dirtscootdesigns5956]

I am currently on that version now and have been since before I noticed the error. Any other ideas?

[Dasbaum]

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.