Error too many open files even after updates

[Dirtscootdesigns5956]

Alert from DuVall-Backup:

Failed to check for alert SyslogNg: [Errno 24] Too many open files

 

 This is the error I keep getting sent to my email, never had an issue before during about a year of hex os use. Any ideas on how to fix it? I have recently made sure everything is up to date! Thanks! Loving the product so far! So worth the lifetime licence!

[Dasbaum]

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.

[mill3000]

Thanks for posting what you found. I informed a team member to review this. If everything is in order, they will submit a ticket to IX (TrueNAS) for the next system update. We appreciate all the members helping out to make HexOS better for everyone. One of the postives of the internet and forums that don't get appriciated enough.

[DubTee]

On 3/13/2026 at 3:23 AM, Dasbaum said:

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.

Genius!  Thank you so much for this, genuinely.  Whatever this bug is has been preventing me from upgrading for months.  I'd check updates once in a while and then just regress and go back to running 25.04.  As soon as I hit ENTER key on the last line of the script, everything settled down immediately.  

I think you can tell at what time the script was entered lol.

[mill3000]

Hello,

One way that could fix this is to do the following. 

Update to TrueNAS version 25.10.2.1. which applies permanent resource limit fixes.  

Thank you

[Dirtscootdesigns5956]

I am currently on that version now and have been since before I noticed the error. Any other ideas?

[Skeleton Man]

On 3/13/2026 at 10:23 AM, Dasbaum said:

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.

It is an interesting fix, an you may be onto something, I am also having infinite middleware processes being spawn over time pushing the file limits, at first I didn't notice the excessive proccesses and just increased the FD Limit, this caused the middleware processes to go rampant and to consume all ram available eventually causing applications to fail.

I tried to apply the fix that you propose but my system failed to start at all and running "systemctl status websocat.service" returns Unit "websocat.service could not be found."
Sadly I am not as knowledgeable on truenas or this level of linux systems, so I am unaware how common this is.

Hopefully a fix can be found soon, for now I will keep the 1024 FD Limit as it keeps my system stable

[Dirtscootdesigns5956]

On 3/15/2026 at 8:40 AM, Skeleton Man said:

It is an interesting fix, an you may be onto something, I am also having infinite middleware processes being spawn over time pushing the file limits, at first I didn't notice the excessive proccesses and just increased the FD Limit, this caused the middleware processes to go rampant and to consume all ram available eventually causing applications to fail.

I tried to apply the fix that you propose but my system failed to start at all and running "systemctl status websocat.service" returns Unit "websocat.service could not be found."
Sadly I am not as knowledgeable on truenas or this level of linux systems, so I am unaware how common this is.

Hopefully a fix can be found soon, for now I will keep the 1024 FD Limit as it keeps my system stable

 

On 3/13/2026 at 4:23 AM, Dasbaum said:

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.

Thank you both for your help! Hope this all gets resolved soon!

[Turbocat777]

On 3/13/2026 at 6:23 AM, Dasbaum said:

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'

 

[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload

 

sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload

 

sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.

Thanks for this, I have been getting these but ignoring them for the most part cause everything was still working and it kept persisting through new updates so I was worried but still ignoring it.

I think I followed the steps correctly, the only thing I ran into was when pasting in the new service I found if you copy a space after "EOF" it doesn't exit correctly and you end up pasting the next commands in incorrectly.

I've attempted to format the steps a big more clearly above. (if it's still wrong feel free to correct me and/or edit your post to clearly show the steps a bit better. (I am def still a novice with TrueNAS and linux/shell commands)

[M4GI3R]

On 3/13/2026 at 11:23 AM, Dasbaum said:

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

I followed the instructions and it solved the problem for me. Can confirm an instant CPU usage drop. Thanks!

[MaLiXs]

On 3/13/2026 at 6:23 AM, Dasbaum said:

I'm having the exact same issue on 25.10.2.1 — so the update alone doesn't fix it. I dug into it and found the actual root cause:

The ix-vendor.service runs /usr/bin/start_vendor_service on boot, which calls vendor_service.py. This script starts a transient systemd unit called websocat.service (the websocket bridge between your local middleware and deck.hexos.com). The problem is that the script doesn't check whether websocat is already running before calling systemd-run --unit=websocat. If websocat is already active, the call fails with "Unit websocat.service was already loaded", but the script always exits with code 0 (it has a finally: sys.exit(0) block).

Something then re-triggers ix-vendor, it fails again, and you get a rapid restart loop.

Each loop iteration spawns a Python process that opens Docker sockets, and since the middlewared service only has a soft file descriptor limit of 1024, the FDs get exhausted quickly — hence the "Too many open files" error.

Here's the fix that worked for me in truenas shell — a systemd override that adds a simple check before running the script:

sudo mkdir -p /etc/systemd/system/ix-vendor.service.d/

sudo tee /etc/systemd/system/ix-vendor.service.d/no-loop.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/bin/bash -c 'if systemctl is-active --quiet websocat.service; then echo "websocat already running, skipping"; exit 0; fi; /usr/bin/start_vendor_service'
EOF

sudo systemctl daemon-reload
sudo systemctl restart ix-vendor.service

After this, ix-vendor shows active (exited) with status 0, websocat keeps running, and deck.hexos.com works fine. No more loop, no more FD exhaustion.

I also raised the middleware FD limit as an extra safety net:

sudo mkdir -p /etc/systemd/system/middlewared.service.d/

sudo tee /etc/systemd/system/middlewared.service.d/fd-limit.conf << 'EOF'
[Service]
LimitNOFILE=65536
EOF

sudo systemctl daemon-reload
sudo systemctl restart middlewared

Both overrides live in /etc/systemd/system/ which - as far as I understand it "so far" - survives reboots. After a HexOS update you may want to verify they're still in place.

Hopefully the HexOS team can add the is-active check to vendor_service.py itself so this gets fixed upstream.

Thanks for the fix! Like other people notice it also reduce cpu usage quite a bit, this gonna help my old i5 4690k based system

 

but like you noted the script does not seem to survive an update cuz I just updated to 25.10.3 from 25.10.2.1 and I had to redo your fix.