Error too many open files even after updates

[Dirtscootdesigns5956]

Alert from DuVall-Backup:

Failed to check for alert SyslogNg: [Errno 24] Too many open files

 

 This is the error I keep getting sent to my email, never had an issue before during about a year of hex os use. Any ideas on how to fix it? I have recently made sure everything is up to date! Thanks! Loving the product so far! So worth the lifetime licence!

[mill3000]

Hello,

One way that could fix this is to do the following. 

Update to TrueNAS version 25.10.2.1. which applies permanent resource limit fixes.  

Thank you

[Dirtscootdesigns5956]

I am currently on that version now and have been since before I noticed the error. Any other ideas?

[Dasbaum]

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.