memory leak in mocord

[fontiano]

I have been trying for a few days to fix a problem with 2 Dahua IPCAMs (which work perfectly in the 1.34 production system) on my test environment with Zoneminder 1.36.33 (Debian 12 - Ryzen 2600x - 16 Gb Ram - SSD).

If I configure them in "Modect" mode everything works fine. If, on the other hand, I configure them in "Mocord" mode, after a few minutes the /usr/bin/zmc -m X processes begin to occupy all the available memory (including swap) and the system becomes unresponsive.

I set the keyframe to 15 on both cameras and set 45 in the "Maximum Image Buffer Size (frames) " parameter to limit the memory usage but it doesn't seem to work.

The only error message I have in the logs is: "[You have set the max video packets in the queue to 45. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval 15 is larger than this setting.]" and appears only sometimes even in "Modect" mode without creating the memory problem.

I tried increasing the buffer to 300 but the problem persists and after a while the system memory runs out.

Anyone have the same problem?

[SenorDeeds]

I have the exact same problem, only running two cameras a Reolink 4k camera and a Bosch 1080p camera both set to Record mode. This only recently started after I did an apt upgrade and apt-get update and rebooted my server, I believe it upgraded my zoneminder to the latest version 1.36.33 and I also believe it did a kernel update. I am running Ubuntu Server 22.04 latest build on an HPE Proliant ML310e Gen 8 V2 server tower.

Now, after maybe 20 minutes or so the server completely runs out of memory (24GB ECC) along with 8 GB of swap, and its all zoneminder. This didn't used to happen, I tried setting frame buffer max to 100 for both cameras and it still leaks out and consumes all the available memory. This crashes the server and even causes the network to drop. I tried lowering the frame rate and bitrate to see if that would help and it doesn't.


I have just the two cameras but 4 monitors, two live feeds set to Monitor and two recording feeds set to Record. At first I thought it was a filter that may be causing some kind of craziness, but I stopped running all filters in the background and the issue persists. I turned off the recording feeds and have been monitoring zoneminder to see if it has a memory leak, and so far it has been stable, memory usage has been relatively stable for over an hour, by this time before it would have crashed already.

So clearly something is going on with the recording features in Zoneminder right now, I didn't change anything about my monitors or the cameras and they WERE taking up a modest amount of memory before with no problems, now suddenly it's never ending memory leaks even when setting frame buffer maximums and tweaking the settings, it doesn't matter.

[SenorDeeds]

Update:

I didnt have any footage I cared to save so I went ahead and deleted and recreated the monitors, kind of inconvenient but now everything appears to be normal again, knock on wood, and has been for a little bit, i'm barely at 1.5 GB of RAM usage now.

One thing I discovered that blew my mind is JUST how much storage space it was taking up by using the JPEG storage option. I went from 10.5 GB at 15 FPS for a 10 minute night video (22 GB for daytime video) from my 4K camera to now 300MB at night! I couldn't believe how much storage it was eating up before this, I just setup the server a few weeks ago and have been playing around with it since then, my 11 TB felt like nothing before this.

Because I am not doing any analysis or modect/mocord or anything, I set "save JPEG" to disabled with video writer set to camera passthrough. This still doesn't necessarily explain the runaway RAM usage as the settings were like that when I set this up and it was stable until I updated. Maybe backup your footage somewhere temporarily and then remake your monitors and see if that works.

[Magic919]

Regardless of analysis it’s generally better to not save JPEGs. Aside from storage, there does seem to be a correlation with increased ram use.

[iconnor]

People don't understand just how much cpu it takes to encode jpegs.

[fontiano]

My settings:

settings.png (195.83 KiB) Viewed 18670 times

Memory usage @ zoneminder start and after 20 minutes:

memory.png (400.29 KiB) Viewed 18670 times

[iconnor]

The only thing out of the ordinary there is the 15 alarm frame count setting...Can't imagine why that would cause a mem leak, but I can try it out here as well.

[fontiano]

The only thing out of the ordinary there is the 15 alarm frame count setting...Can't imagine why that would cause a mem leak, but I can try it out here as well.

Cameras record at 15 fps and I set the alarm frame count to 15 to avoid flapping alarms.

I tried to leave everything at default but the situation does not change

[fontiano]

Update:

I didnt have any footage I cared to save so I went ahead and deleted and recreated the monitors, kind of inconvenient but now everything appears to be normal again, knock on wood, and has been for a little bit, i'm barely at 1.5 GB of RAM usage now.

One thing I discovered that blew my mind is JUST how much storage space it was taking up by using the JPEG storage option. I went from 10.5 GB at 15 FPS for a 10 minute night video (22 GB for daytime video) from my 4K camera to now 300MB at night! I couldn't believe how much storage it was eating up before this, I just setup the server a few weeks ago and have been playing around with it since then, my 11 TB felt like nothing before this.

Because I am not doing any analysis or modect/mocord or anything, I set "save JPEG" to disabled with video writer set to camera passthrough. This still doesn't necessarily explain the runaway RAM usage as the settings were like that when I set this up and it was stable until I updated. Maybe backup your footage somewhere temporarily and then remake your monitors and see if that works.

I just tried to delete and reconfigure monitors without success. As you can see from settings I have not enabled jpeg saving.

[fontiano]

I moved HD to new HW: Xeon1230 V5 - 16 Gb ram
but I've got same problem

memory.png (43.17 KiB) Viewed 18596 times

sometimes this appears in the logs (zmc_m4 and zmc_m3)

2023-07-15T16:13:17.760514+02:00 zoneminder136 zmc_m4[1702]: WAR [zmc_m4] [You have set the max video packets in the queue to 100. The queue is full. Either Analysis is not keeping up or your camera's keyf>
2023-07-15T16:13:17.760650+02:00 zoneminder136 zmc_m4[1702]: WAR [zmc_m4] [Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up.]

[fontiano]

I unchecked "Analysis Enabled" but I've got the same problem. I also tried to add a third camera and this is the result:

memory.png (49.28 KiB) Viewed 18593 times

[fontiano]

Same settings, 3 cameras but modect after 1 hour:

mem.png (40.59 KiB) Viewed 18589 times

I think that's a problem with continuous recording not related to video analysis

[dougmccrary]

Are you recording sound?

[burger]

Mocord/dect on a 2.5k pixel stream? I would only do modect or mocord on the low res stream. Try linked monitors or run two monitors (low / hi res stream w/only record on the hi res stream).

You could also try changing analysis fps to 2 or 3.

[fontiano]

Are you recording sound?

no sound recording