Page 6 of 6
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Sat Aug 27, 2022 3:02 am
by fvalley2
Hey guys,
I see others still having this issue and wanted to post my own resolution/issue.
Previously worked around memory/swap over filling by switching off camera passthrough for all cams. Today I noticed that one of my cams, an Amcrest, had "smart codec" on in the cams onboard setup. Switching this off seems to have fixed the issue for me. Turning it on again repeated. Off and everything is fine.
In hindsight, htop was showing this camera was soaking up all the extra memory all along. Apologies for any confusion.
Hope this helps someone.
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Thu Sep 08, 2022 6:13 am
by slipperyduck
I've also been experiencing the same issue (as likely most of us are) with the "memory leak" and I think i have finally actually found the culprit.
I am no expert but there is something not flushing with EVENTS.
I use [record] mode only, no analysis, just straight recording. I have 15 cameras, so the "memory leak" as it has been labelled increases quite rapidly.
Luckily I've got a beefy server, 18cores/36 thread and 256Gb or Memory.
I found that the higher the bit rate of the cameras the faster the memory gobbling takes place.
Now this morning I decided to purge all events (recordings) after setting all my cameras to a good balance of lower-bitrate vs image quality.
The PURGING of the events completely flushed all memory usage, so I waited an hour, watch the memory usage climb and then purged the events again, indeed the same thing happened, all memory usage was "reset".
*all my camera monitors are passthough FFMPG sources using rtsp
Could it be that the event system that is not closing / ending its process resulting in each new event start chomping up a new chunk of memory until memory runs out?
ZM v1.36.25 (debian)
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Thu Sep 08, 2022 5:29 pm
by iconnor
While anything is possible, I simply don't see this on any of my boxes.
When saving events, because it can take some time to finalise an mp4, we actually hand off control to another thread, and start a new thread to write the next event. So there is a small potential of a leak but I've looked very closely at this code for a while now. If your drives were so slow that writing to disk is slower than capture, then the packetqueue will fill up and you will get complaints about it. Are you getting complaints about the packetqueue being full? Do you have a MaxImageBuffer set?
I'm trying to recreate this here, so I've setup a monitor in record only, passthrough, no audio, section length=30 (so that we get lots of events). So far after running for hours, mem use is still bouncing around 8%.
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Fri Sep 09, 2022 5:44 am
by slipperyduck
Hi, thanks for looking in to this, I eventually figured out my case, which is a fresh install and update of debian with the apt package install for everything.
after so much digging and troubleshooting I found my case was basically the Mariadb not having enough buffer memory
my fix was this:
edited /etc/mysql/conf.d/mysql.cnf
Added:
Innodb_buffer_pool_size 16G
restarted mariadb service
systemctl restart mysql
Now my memory usage hovers around a meger 5Gb and everything is working great.
Really appreciate all the effort and engagement you put in iconnor!
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Thu Sep 15, 2022 7:22 pm
by zSprawl
After 8 days of running on Oracle Linux 8, I crashed with out of memory errors galore on the screen. The cpu was pegged at 100% and I could barely initiate a reboot much less look at logs.
My innodb was set to 128M so I bumped it up to 2G mostly cause I have no clue what else to do.
@iConnor - what OS do you build your test labs on? I’m thinking I just copy your deployment and avoid these issues, if possible.
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Sun Apr 16, 2023 2:59 am
by 166MHz
I have the same problem with memory leak in zmc.
I noticed that not all cameras leak and tested how the settings differ and found that leaks occur only on cameras where audio stream recording is enabled! If I turn off audio recording, everything works without leaks in zmc
At the same time, it does not matter where to turn off the stream, in zm or on the camera itself, if it is turned off, there is no memory leak. I also tested several audio encoding options on the camera, with any of the tested ones, I see a memory leak. I'm still continuing to test in this way, but I can't say for sure yet that there is a leak on all audio codecs
my env:
2 srvrs with Debian 11 / ZM 1.36.33 / Mysql 8.0.32
Over 60 cameras Hikvision (different models) with settings / RTSP / H264+ or H264 / Camera Passtrought / buffers ImageBufferCount=3 MaxImageBufferCount= 50-250 / Mode "Record" / Analyze OFF/
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Thu Apr 20, 2023 2:37 pm
by whoare365q
I have tweaked a bit the setup and removed one camera at the same time, so I can't be completely sure what is the driver of the improvement on my system.
pikashow.fyi
ppssppgold.one
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Fri Apr 21, 2023 10:04 pm
by iconnor
Would be interesting to run one of your afflicated cams under valgrind to see if it reports anything.
The audio code is separate and not as well tested, as most people don't record the audio. Also there are two code paths, if the camera provides AAC then it's a passthrough situation, otherwise we have to encode it to AAC. More work, more ram, more load on disks, etc.
So if you install valgrind, and shutdown a monitor, let's say number 1. Then you can run it from command line using something like:
sudo su -s/bin/bash -c"valgrind --leak-check=yes zmc -m1" www-data
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Mon Apr 24, 2023 7:03 am
by Viher
I set the monitor to None on ZM and ran the command. Reolink RLC-810A with audio enabled for this test.
Code: Select all
04/24/23 09:45:46.084845 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:46:20.079561 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:47:00.092816 zmc_m2[87268].ERR-zm_ffmpeg_camera.cpp/231 [Unable to read packet from stream 0: error -5 "Input/output error".]
04/24/23 09:47:00.154933 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:47:50.100861 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:48:34.098426 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:49:20.075121 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:50:05.098781 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:51:04.084777 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:51:44.370044 zmc_m2[87334].ERR-zm_ffmpeg.cpp/518 [Unable to send packet Invalid data found when processing input, continuing]
04/24/23 09:51:44.371568 zmc_m2[87334].WAR-zm_packet.cpp/122 [Unable to receive frame : code -1094995529 Invalid data found when processing input.]
04/24/23 09:52:17.126304 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:53:16.080963 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:54:14.121251 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:55:06.115609 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:55:56.107497 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:56:51.093295 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:57:49.124871 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:58:24.122439 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 09:59:12.151616 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:00:01.078429 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:00:48.107174 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:01:33.156249 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:02:13.103677 zmc_m2[87268].ERR-zmc.cpp/305 [Failed to capture image from monitor 2 Autoparkki (1/1)]
04/24/23 10:04:04.031857 zmc_m2[87268].ERR-zm_ffmpeg_camera.cpp/231 [Unable to read packet from stream 0: error -5 "Input/output error
I'm going to let it run for couple of hours at least, to see if it gives something different than those lines above.
EDIT:
No lines different than those above after 3 hours.
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Mon Apr 24, 2023 1:12 pm
by iconnor
So the output comes once you stop it. It should give a summary of how much ram was leaked.
Those errors are not good though, do they happen when you don't run it under valgrind?
valgrind makes things run a lot slower due to all it's memory accounting... so it might just be that you aren't able to keep up anymore.
I might also suggest that you turn on debugging.
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Tue Apr 25, 2023 5:19 am
by Viher
Code: Select all
Session terminated, killing shell...==87268==
==87268== HEAP SUMMARY:
==87268== in use at exit: 51,484 bytes in 275 blocks
==87268== total heap usage: 2,687,792 allocs, 2,687,517 frees, 527,293,872,182 bytes allocated
==87268==
==87268== LEAK SUMMARY:
==87268== definitely lost: 0 bytes in 0 blocks
==87268== indirectly lost: 0 bytes in 0 blocks
==87268== possibly lost: 0 bytes in 0 blocks
==87268== still reachable: 49,468 bytes in 254 blocks
==87268== suppressed: 0 bytes in 0 blocks
==87268== Reachable blocks (those to which a pointer was found) are not shown.
==87268== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==87268==
==87268== For lists of detected and suppressed errors, rerun with: -s
==87268== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
...killed.
I'll turn on debugging on and see if those errors come up. Log now shows only warnings or worse, at this level those errors don't turn up.
Re: Memory Leak/Increase and Choppy events due to Maximum Image Buffer Size
Posted: Fri May 05, 2023 2:23 pm
by tinymouse
I have several TAPO C310 cams and only one of them has increasing ram usage; usually grows daily. Starts off at 3gb, but then over a few days it grows to well over 16gb before I have to restart it. At first I thought it was being in RECORD mode, or setting at 2k (2304x1296). Nope, none of those. Then reading the response from
166MHz about
audio stream recording made me look at my cam and behold, I too have the "audio stream when saving an event" selected. I un-selected this option; restarted that zmc and now.... all is normal with this cam. It's using between 4gb and 6gb of ram for the past couple of weeks.
Big Thanks to 166MHz for mentioning this.
166MHz wrote: ↑Sun Apr 16, 2023 2:59 am
I have the same problem with memory leak in zmc.
I noticed that not all cameras leak and tested how the settings differ and found that leaks occur only on cameras where
audio stream recording is enabled! If I turn off audio recording, everything works without leaks in zmc