The memory fills progressively

[fontiano]

Debian 12 - Cpu: Ryzen 5 PRO 3400G - Ram: 32 Gb - SSD for SO and DB - HDD for events recording

After starting Zoneminder the memory occupied is approximately 3 Gb

After 6 minutes 8.44 Gb:

after-6-minutes.png (119.78 KiB) Viewed 19118 times

After 21 minutes 17.1 Gb:

after-21-minutes.png (98.68 KiB) Viewed 19118 times

After 57 minutes 32 Gb of ram and all the swap space:

after-57-minutes.png (112.32 KiB) Viewed 19118 times

All the cams are configured in the same mode:
Capturing: Always
rtsp://
Decoder: auto
Decoding: Keyframes + Ondemand
Motion Detection: Always
Analysy Image: Y-CHannel
Analysis FPS: 15
Video Writer: Camera Passthrough
SDave JPEGs: Analysis image only

[mikb]

How many cameras do you actually have?

The count of tasks in those screen shots seems to be climbing too, is it possible you have more capture processes than you have cameras (e.g. something is causing them to stall, so ZM launches a new capture process, and again, and again, for every camera) ?

That would certainly cause you some problems.

[fontiano]

No, that's the htop view.

Look @ top view:
Start:

start.png (81.65 KiB) Viewed 18758 times

After some minutes:

after-some-minutes.png (69.35 KiB) Viewed 18758 times

In one hour no more memory available.

EDIT:

You are right, I forgot to order by mem usage:

proces.png (118.86 KiB) Viewed 18738 times

[fontiano]

I reproduced the problem with a single camera in a VMWare machine:

Settings I'm using:

recording.png (14.5 KiB) Viewed 18750 times

misc.png (17.08 KiB) Viewed 18750 times

buffers.png (12.63 KiB) Viewed 18750 times

[fontiano]

How many cameras do you actually have?

The count of tasks in those screen shots seems to be climbing too, is it possible you have more capture processes than you have cameras (e.g. something is causing them to stall, so ZM launches a new capture process, and again, and again, for every camera) ?

That would certainly cause you some problems.

Yes, you found the problem.

11 cameras

root@zoneminder137:/usr/bin# date
mar 16 gen 2024, 18:19:54, CET
root@zoneminder137:/usr/bin# ps auxww | grep zmc | wc -l
53
root@zoneminder137:/usr/bin# date
mar 16 gen 2024, 18:20:01, CET
root@zoneminder137:/usr/bin# ps auxww | grep zmc | wc -l
57
root@zoneminder137:/usr/bin# date
mar 16 gen 2024, 18:20:10, CET
root@zoneminder137:/usr/bin# ps auxww | grep zmc | wc -l
57
root@zoneminder137:/usr/bin# date
mar 16 gen 2024, 18:20:27, CET
root@zoneminder137:/usr/bin# ps auxww | grep zmc | wc -l
58
root@zoneminder137:/usr/bin# date
mar 16 gen 2024, 18:20:43, CET
root@zoneminder137:/usr/bin# ps auxww | grep zmc | wc -l
60

[mikb]

OK, so at least you know what exactly you are looking to fix now

Your capture processes (taking data from the camera) are not starting and running continuously. They are instead getting hung up, so ZM sees this as a fault and starts a new one -- it's persistent in trying to keep your security system running. Repeat until memory all gone.

With a single camera it would be easier to fault find -- check the ZM logs to see if it gives a clue why the zmc process isn't happy. E.g. things like analysis hasn't had a new frame for a while, or network errors causing disconnections of the camera etc.

[iconnor]

Zm is supposed to kill processes not leave them around... This is pretty weird

[fontiano]

If I launch "systemctl stop zoneminder" the command does not finish and remains hanging waiting

To make it terminate I have to kill the zmc processes with "killall -9 zmc"

I have the same problem with the same settings (with 1 cam only) in a separate test virtual machine.

[iconnor]

Look in your logs. The answer will be there.

[fontiano]

Ok, I don't know if it can be called a bug but I have identified the problem.

I had replicated the configurations identically and the zmc processes remain hanging when the command entered on "Event End Command" fails.

A simple "echo $1" generates the anomaly.



To this day I still haven't figured out how to make it work.

[iconnor]

I recently fixed the problem with Event End Command. Please update.

[marklg]

It seems that zoneminder-1.37.50-1.38.20240122gitg5cc7ca28e.fc39.x86_64 still has the issue. Instances of zmc progressively take up more memory. In a few hours, it runs out of RAM and starts using Swap. Restarting Zoneminder frees up the memory.

Edit - A while after the last restart, memory usage jumped down but the overall trend it upward, with occasional small jumps down.

Regards,

Mark

[iconnor]

I have seen many times, including lately on my own server, where slow disk writes cause this. Currently on my home server, which has been chugging along great for years, one of the drives seems to be going. SMART still says it's ok, but throughput is dropping and occasionally the raid card spits it out .

Removing it from the storage pool solves it.

Did an investigation with someone having the same issue with a brand new 18TB drive. Turns out it was an SMR drive. Shingled Magnetic Recording, on which writes are very slow.

So that is always my suspicion. Because I have spent a LOT of time trying to diagnose this over the years and I manage a lot of servers that do not have this problem.

It seems ridiculous to me though that any modern hard drive can't keep up to what is really not that much writing. If the network can keep up, the disk should be able to as well.

[marklg]

I am very careful to not buy SMR drives. I only have CMR and decent SSDs. This issue was not present on versions from a week or so ago.

For now, I have this in my crontab. It seems to trigger every couple hours.

5 * * * * root if [ $(grep MemFree /proc/meminfo | sed "s/[^0-9]//g") -lt 400000 ]; then /usr/bin/systemctl restart zoneminder; fi

Regards,

Mark

[iconnor]

So.. I just found a memory leak when reloading config after a HUP. How this has escaped notice, I have NO idea. So this would have leaked 22k per camera every night.