Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size

[fvalley2]

Hey guys,

I see others still having this issue and wanted to post my own resolution/issue.

Previously worked around memory/swap over filling by switching off camera passthrough for all cams. Today I noticed that one of my cams, an Amcrest, had "smart codec" on in the cams onboard setup. Switching this off seems to have fixed the issue for me. Turning it on again repeated. Off and everything is fine.

In hindsight, htop was showing this camera was soaking up all the extra memory all along. Apologies for any confusion.

Hope this helps someone.

[slipperyduck]

I've also been experiencing the same issue (as likely most of us are) with the "memory leak" and I think i have finally actually found the culprit.
I am no expert but there is something not flushing with EVENTS.
I use [record] mode only, no analysis, just straight recording. I have 15 cameras, so the "memory leak" as it has been labelled increases quite rapidly.
Luckily I've got a beefy server, 18cores/36 thread and 256Gb or Memory.
I found that the higher the bit rate of the cameras the faster the memory gobbling takes place.

Now this morning I decided to purge all events (recordings) after setting all my cameras to a good balance of lower-bitrate vs image quality.
The PURGING of the events completely flushed all memory usage, so I waited an hour, watch the memory usage climb and then purged the events again, indeed the same thing happened, all memory usage was "reset".
*all my camera monitors are passthough FFMPG sources using rtsp

Could it be that the event system that is not closing / ending its process resulting in each new event start chomping up a new chunk of memory until memory runs out?

ZM v1.36.25 (debian)

[iconnor]

While anything is possible, I simply don't see this on any of my boxes.

When saving events, because it can take some time to finalise an mp4, we actually hand off control to another thread, and start a new thread to write the next event. So there is a small potential of a leak but I've looked very closely at this code for a while now. If your drives were so slow that writing to disk is slower than capture, then the packetqueue will fill up and you will get complaints about it. Are you getting complaints about the packetqueue being full? Do you have a MaxImageBuffer set?

I'm trying to recreate this here, so I've setup a monitor in record only, passthrough, no audio, section length=30 (so that we get lots of events). So far after running for hours, mem use is still bouncing around 8%.

[slipperyduck]

Hi, thanks for looking in to this, I eventually figured out my case, which is a fresh install and update of debian with the apt package install for everything.
after so much digging and troubleshooting I found my case was basically the Mariadb not having enough buffer memory
my fix was this:

edited /etc/mysql/conf.d/mysql.cnf
Added:
Innodb_buffer_pool_size 16G

restarted mariadb service
systemctl restart mysql

Now my memory usage hovers around a meger 5Gb and everything is working great.
Really appreciate all the effort and engagement you put in iconnor!

[zSprawl]

After 8 days of running on Oracle Linux 8, I crashed with out of memory errors galore on the screen. The cpu was pegged at 100% and I could barely initiate a reboot much less look at logs.

My innodb was set to 128M so I bumped it up to 2G mostly cause I have no clue what else to do.

@iConnor - what OS do you build your test labs on? I’m thinking I just copy your deployment and avoid these issues, if possible.

[166MHz]

I have the same problem with memory leak in zmc.

I noticed that not all cameras leak and tested how the settings differ and found that leaks occur only on cameras where audio stream recording is enabled! If I turn off audio recording, everything works without leaks in zmc

At the same time, it does not matter where to turn off the stream, in zm or on the camera itself, if it is turned off, there is no memory leak. I also tested several audio encoding options on the camera, with any of the tested ones, I see a memory leak. I'm still continuing to test in this way, but I can't say for sure yet that there is a leak on all audio codecs

my env:
2 srvrs with Debian 11 / ZM 1.36.33 / Mysql 8.0.32
Over 60 cameras Hikvision (different models) with settings / RTSP / H264+ or H264 / Camera Passtrought / buffers ImageBufferCount=3 MaxImageBufferCount= 50-250 / Mode "Record" / Analyze OFF/

[whoare365q]

I have tweaked a bit the setup and removed one camera at the same time, so I can't be completely sure what is the driver of the improvement on my system.
pikashow.fyi
ppssppgold.one

[iconnor]

Would be interesting to run one of your afflicated cams under valgrind to see if it reports anything.

The audio code is separate and not as well tested, as most people don't record the audio. Also there are two code paths, if the camera provides AAC then it's a passthrough situation, otherwise we have to encode it to AAC. More work, more ram, more load on disks, etc.

So if you install valgrind, and shutdown a monitor, let's say number 1. Then you can run it from command line using something like:
sudo su -s/bin/bash -c"valgrind --leak-check=yes zmc -m1" www-data

[Viher]

I set the monitor to None on ZM and ran the command. Reolink RLC-810A with audio enabled for this test.

Code: Select all

04/24/23 09:45:46.084845 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:46:20.079561 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:47:00.092816 zmc_m2[87268].ERR-zm_ffmpeg_camera.cpp/231 [Unable to read packet from stream 0: error -5 "Input/output error".]
04/24/23 09:47:00.154933 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:47:50.100861 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:48:34.098426 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:49:20.075121 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:50:05.098781 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:51:04.084777 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:51:44.370044 zmc_m2[87334].ERR-zm_ffmpeg.cpp/518 [Unable to send packet Invalid data found when processing input, continuing]
04/24/23 09:51:44.371568 zmc_m2[87334].WAR-zm_packet.cpp/122 [Unable to receive frame : code -1094995529 Invalid data found when processing input.]
04/24/23 09:52:17.126304 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:53:16.080963 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:54:14.121251 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:55:06.115609 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:55:56.107497 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:56:51.093295 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:57:49.124871 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:58:24.122439 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:59:12.151616 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:00:01.078429 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:00:48.107174 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:01:33.156249 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:02:13.103677 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:04:04.031857 zmc_m2[87268].ERR-zm_ffmpeg_camera.cpp/231 [Unable to read packet from stream 0: error -5 "Input/output error

I'm going to let it run for couple of hours at least, to see if it gives something different than those lines above.

EDIT:
No lines different than those above after 3 hours.

[iconnor]

So the output comes once you stop it. It should give a summary of how much ram was leaked.

Those errors are not good though, do they happen when you don't run it under valgrind?

valgrind makes things run a lot slower due to all it's memory accounting... so it might just be that you aren't able to keep up anymore.

I might also suggest that you turn on debugging.

[Viher]

Code: Select all

Session terminated, killing shell...==87268==
==87268== HEAP SUMMARY:
==87268==     in use at exit: 51,484 bytes in 275 blocks
==87268==   total heap usage: 2,687,792 allocs, 2,687,517 frees, 527,293,872,182 bytes allocated
==87268==
==87268== LEAK SUMMARY:
==87268==    definitely lost: 0 bytes in 0 blocks
==87268==    indirectly lost: 0 bytes in 0 blocks
==87268==      possibly lost: 0 bytes in 0 blocks
==87268==    still reachable: 49,468 bytes in 254 blocks
==87268==         suppressed: 0 bytes in 0 blocks
==87268== Reachable blocks (those to which a pointer was found) are not shown.
==87268== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==87268==
==87268== For lists of detected and suppressed errors, rerun with: -s
==87268== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
 ...killed.

I'll turn on debugging on and see if those errors come up. Log now shows only warnings or worse, at this level those errors don't turn up.

[tinymouse]

I have several TAPO C310 cams and only one of them has increasing ram usage; usually grows daily. Starts off at 3gb, but then over a few days it grows to well over 16gb before I have to restart it. At first I thought it was being in RECORD mode, or setting at 2k (2304x1296). Nope, none of those. Then reading the response from 166MHz about audio stream recording made me look at my cam and behold, I too have the "audio stream when saving an event" selected. I un-selected this option; restarted that zmc and now.... all is normal with this cam. It's using between 4gb and 6gb of ram for the past couple of weeks.
Big Thanks to 166MHz for mentioning this.

I have the same problem with memory leak in zmc.

I noticed that not all cameras leak and tested how the settings differ and found that leaks occur only on cameras where audio stream recording is enabled! If I turn off audio recording, everything works without leaks in zmc