New system, RAM problem

[Viher]

Hi,

I just upgraded my server hardware to accommodate few 4k cameras and made a fresh install on both OS and ZM. Been using zoneminder quite a long time now, over 6 years and now ran into severe problem hindering the servers ability to even record stable footage.

Server:
Intel i9-13900k, 64GB RAM, about 21TB of space on 7 HDDs and one fast SSD gathered up in LVM.
Ubuntu server 22.04
ZM from iconnors PPA, version installed 1.36.33

Cameras added to ZM (couple more waiting to be installed and added after this is solved):

2 Reolink 810A 4k @20fps 8000kbit/s
1 Unifi UVC G3 1080p @20fps 5000kbit/s
1 AXIS Companion Eye LVE 1080p @20fps 5000kbit/s

All cameras on RECORD, without analysis or decoding on pass-through, max image queu set on all cameras. Live view still works on all, even though system threatens it would be unavailable with those settings.

Everything goes smoothly for about 30min after zm restart, total RAM usage in 6-8gb. Then suddenly in under one minute, RAM usage tops to 64GB (64GB SWAP fills in 10 secs when RAM usage hits about 62GB) and system load goes from 5 to 20-50 for a while, all CPU cores max for a minute or two and then system kills zmc with out of memory message. Memory usage drops to normal and usually immediately starts to build up again and above mentioned behavior keeps repeating. (now as I write this memory use stayed in 33 GB for 5min and then maxed out again).

Htop on server tells that reolink cameras are using all the memory with 4 processes like other two also, log gives error on other two about image queue being full when memory hoarding from reolinks has started.

Log on debug level atm, but gives nothing before memory is about to run out.

Tried searching web for two days, but did not get very far, mainly few year old topics about memory leaks.

Quite tired at the moment, probably forgot to mention lots of things.

[Viher]

I don't frequently visit this forum and it seems that caused me to miss quite a lot of useful conversations.
Good thing few useful topics popped up, for example viewtopic.php?p=130269#p130269

Information seems to scattered around and my search terms need some adjusting, but I gathered some things to try:

Audio is now disabled on all streams
line "reorder_queue_size=1000" added to reolink cameras source->options, even though it was not deemed necessary on RLC-810A, wiki only recommended it to 820A
In Mysql conf file i've set: Innodb_buffer_pool_size=8G and will be doing other things recommended by mysqltuner, after it has been running enough.
Save JPEGs set to disabled. This killed the live streams as expected, had misread when this would happen.

But memory use still keeps getting up to a point system has to kill processes to stay functional, viewing servers resource use constantly via htop in my other monitor, I noticed that memory caching slowly builds up to max, before real hoarding starts.

At least ZM thinks Reolink streams stop working quite frequently, causing recordings to be shorter than expected 10mins or oddly longer that said setting for example 17mins. Viewing stream from vlc shows it to be stable. Zoneminder show odd fps numbers for reolink cams, they are both set to 20fps, which they have been until now.

zm.png (59.51 KiB) Viewed 3371 times

[Magic919]

Usual advice is to not save JPEGs, tune the DB, lower framerates and consider doing analysis on lower res stream (where cameras can supply two streams). Looks like you have no analysis, but presume that is temporary.

I run analysis FPS at 5 and turned off all the DB logging, which felt like it helped. Some have the DB on SSD. Mine auto restarts just once a day.

Forum is rubbish these days due to lack of ZM community. There is some presence on Slack and apparently Discord too.

[Viher]

Usual advice is to not save JPEGs, tune the DB, lower framerates and consider doing analysis on lower res stream (where cameras can supply two streams). Looks like you have no analysis, but presume that is temporary.

I run analysis FPS at 5 and turned off all the DB logging, which felt like it helped. Some have the DB on SSD. Mine auto restarts just once a day.

Forum is rubbish these days due to lack of ZM community. There is some presence on Slack and apparently Discord too.

Thanks for the reply.

I would like to do analyzing and MOCORD, like i did with previous hardware(without 4k cameras), but that has to wait until there's stable recording of streams.

I disabled the database logging. According to log, system is in green, normal state, until it runs out of memory. Before that everything is fast, opening 500 lines of log, viewing stream records etc. nothing seems to take "no time at all" even when the RAM hoarding is in effect but not in max, load avarage on 32-thread system just above 3. So there seems to and should be plenty of power/overhead for the system to be able to record streams, or more.

[Viher]

After I realized I made some stupid choices when setting up LVM (included 980 pro in it, rather than leaving it solely for OS to use), I reinstalled OS and ZM.

Now load average seems to be around 0.01-0.12 with above mentioned streams on RECORD with all the changes I previously made to streams. Has been stable for 60min now, I hope it stays that way so that I can setup rest of cameras and MOCORD.'

Edit:

Still going after 12 hours, streams have been recorded without interruptions whole time.