Memory leak after update version zoneminder at ubuntu 24.04 amd64

[ipelykh]

Hello ALL.

After updating the version of zm, installed from ppa.launchpadcontent.net/iconnor/zoneminder-master/ubuntu noble, the memory started leaking.
I am testing ubuntu 24.04 with three 8MP cameras (4k image, Hiseeu HB918-PA, ffmpeg source), 10FPS, 1 sec keyframe with a maximum number of frames in the buffer 20 at each. MoDect, Analysis always. Recording passthrow.
Before the update (with version 1.37.65e20250102141650-noble), all was fine and each zmc process consumed no more than 5 GB of memory,
now with version 1.37.65e20250128172246-noble, the memory consumed by zmc processes is constantly growing (10..15..20 GB) quickly after the start zm indefinitely, until all available machine memory runs out, including swap.
I was just doing an apt-get upgrade. Maybe I upgraded incorrectly? At log all is good, no err and war messages.
I am ready give anything that is needed.
I have do test run with full debug (9 level) and log-file is about 0,9Gb. I don't know where is put it. I can give a http-link at PM or email if needed.
Thanks for your answers.

[dougmccrary]

Maybe I upgraded incorrectly?

Perhaps. FWIW, 1.37.65~20250128172246-jammy seems to be OK.

[busymind]

I think I may have the same issues. R730, Proxmox, VM Debian, 12 ZoneMinder. For start/evaluation I had given a lot of resources

The behaviour is like an memory leak. No errors. Innodb increased to 8G. Run smoothly.
At start was - I do not remember - after 11h runtime is beyond 10G.

Attached are the settings most of them as default on buffer. I had the same behaviour with decoding - Always.
I discovered recently the option onDemand liberates resources a lot so I tough that may alleviate the memory leak but continues to growing. I will try to see what will happen when.

Excuse my improper presentation may have to follow some bug reporting format.
I am also a small developer traumatized by memory leaks on my lifetime coding ... so I sense the same here.

[busymind]

Update:
After 21 hours, seems that now is calm and steady; From16G available stays and 14.39G memory without any impact.

[IgorA100]

From16G available stays and 14.39G memory without any impact.

You are looking at the memory size allocated to the virtual machine, this is normal.
You should monitor the memory consumed by ZM processes on the virtual machine itself, for example with the command

Code: Select all

htop

[busymind]

All fine indeed, stable as rock for now. I am running it as VM and I am still new to this. I switch from bare metal.

[ipelykh]

You are looking at the memory size allocated to the virtual machine, this is normal.
You should monitor the memory consumed by ZM processes on the virtual machine itself, for example with the command

Code: Select all

htop

Hello.
Thanks for your reply.
At the moment, I am testing a new system on xeon e5 2680 v4 (14 cores \ 28 threads) with 64GB of RAM.
Zoneminder version 1.37.65~20250219085701-noble. (The latest from the Ubuntu 24.04 Master repository).
So far, there are only 3 cameras 8MP (3840x2160 4K UHD) h265 (hevc) 10 FPS keyframes = 1 sec.
Monitors 1,2,3 - high resolution, without analysis, maximum image buffer size (frames) is unlimited (0), trigger passthrough recording from a low-resolution stream (monitors 4,5,6 (800x448)).
As you can see, the memory consumption of the ZM process is clearly not normal, imho: a bug in the code.
If you fix the buffer, the memory consumption may increase 4-5 times more than the calculated one, while the logs contain a message about buffer overflow. Some times when memory is over I see this:

Code: Select all

user@server:~$ sudo dmesg | grep Killed
[493482.725856] Out of memory: Killed process 294781 (zmc) total-vm:41485776kB, anon-rss:38705796kB, file-rss:4928kB, shmem-rss:323904kB, UID:33 pgtables:80544kB oom_score_adj:0
[556327.151763] Out of memory: Killed process 310082 (zmc) total-vm:39848000kB, anon-rss:24750024kB, file-rss:4704kB, shmem-rss:259168kB, UID:33 pgtables:77216kB oom_score_adj:0
[557462.568729] Out of memory: Killed process 319836 (zmc) total-vm:35275864kB, anon-rss:22990044kB, file-rss:5152kB, shmem-rss:259168kB, UID:33 pgtables:68084kB oom_score_adj:0
[558956.660618] Out of memory: Killed process 320922 (zmc) total-vm:39308412kB, anon-rss:33386476kB, file-rss:3584kB, shmem-rss:259168kB, UID:33 pgtables:76212kB oom_score_adj:0
[560276.441614] Out of memory: Killed process 321380 (zmc) total-vm:51673960kB, anon-rss:35617180kB, file-rss:6048kB, shmem-rss:259168kB, UID:33 pgtables:99384kB oom_score_adj:0
[561317.361425] Out of memory: Killed process 321779 (zmc) total-vm:55183100kB, anon-rss:47625036kB, file-rss:3584kB, shmem-rss:259168kB, UID:33 pgtables:107468kB oom_score_adj:0
[561926.576893] Out of memory: Killed process 322198 (zmc) total-vm:36261528kB, anon-rss:17581704kB, file-rss:4928kB, shmem-rss:259168kB, UID:33 pgtables:62972kB oom_score_adj:0
[562685.900250] Out of memory: Killed process 322416 (zmc) total-vm:35545504kB, anon-rss:23950588kB, file-rss:5152kB, shmem-rss:258944kB, UID:33 pgtables:68244kB oom_score_adj:0
[837524.334692] Out of memory: Killed process 454280 (zmc) total-vm:21680764kB, anon-rss:16092824kB, file-rss:1568kB, shmem-rss:323904kB, UID:33 pgtables:32956kB oom_score_adj:0
[1179110.793518] Out of memory: Killed process 700861 (zmc) total-vm:31134672kB, anon-rss:25182828kB, file-rss:6496kB, shmem-rss:194208kB, UID:33 pgtables:60268kB oom_score_adj:0

Can some look into code, why so much memory usage?

PS after restart ZM normal mem usage is about 1GB of one high res monitor.

[ipelykh]

Searching for similar problems revealed this:
https://github.com/ZoneMinder/zoneminder/issues/3772
In the last comment, dear iconnor writes that the problem has been solved, but apparently not.
The question is open, why is the memory leaking?
And what should be done to launch this system with 8 cameras?

[iconnor]

Your cameras are too high res. Your system cannot keep up. You have not set a MaxImageBuffer limit.

[IgorA100]

ipelykh
You are probably experiencing high memory consumption when viewing in real time.
I recommend using RTSP2Web type WebRTC or MSE and H264 for viewing in real time. It is advisable to set the resolution on the second stream for viewing to no more than FHD, otherwise there will be a large load on the browser when viewing several cameras on the Montage page.