The memory fills progressively

Current Development version likely to have breaking changes
marklg
Posts: 23
Joined: Sun Dec 17, 2023 1:43 am

Re: The memory fills progressively

Post by marklg »

This is much larger, occupying gigs in a few hours. There are very few events and it only records when an event happens.

In the logs, there are a few of these errors:
Error writing packet: Invalid argument

And a few of these warnings:
Telemetry server returned HTTP POST error code: 301

Anything else to look for?

Regards,

Mark
dougmccrary
Posts: 1322
Joined: Sat Aug 31, 2019 7:35 am
Location: San Diego

Re: The memory fills progressively

Post by dougmccrary »

Telemetry server returned HTTP POST error code: 301
There's no server there at present, so turn off Options->Telemetry send checkbox.
User avatar
iconnor
Posts: 3197
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: The memory fills progressively

Post by iconnor »

The Error writing packet: Invalid argument is very very bad. We need debug level 3 logs.
marklg
Posts: 23
Joined: Sun Dec 17, 2023 1:43 am

Re: The memory fills progressively

Post by marklg »

With zoneminder-1.37.50-1.86.20240126gitg543d3bdd9.fc39.x86_64, I still need to restart to free up memory every few hours. I am not seeing recent errors or warnings except for the telemetry server. I have not disabled that setting. Record durations also seem to be correct. It no longer gets any ONVIF events from my "problem" cameras. I expect they do not follow the standard correctly.

Regards,

Mark
dougmccrary
Posts: 1322
Joined: Sat Aug 31, 2019 7:35 am
Location: San Diego

Re: The memory fills progressively

Post by dougmccrary »

It doesn't handle sources that use this sort of theing in 1.36.x either...

rtsp://192.168.1.224/user=admin_password=passwd_channel=0_stream=1.sdp?real_stream
User avatar
iconnor
Posts: 3197
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: The memory fills progressively

Post by iconnor »

@dougmccrary what?

That's not a valid url. I went googling for the valid equivalent and found tons of problems on the way. Maybe tomorrow I can be of more assistance.

but the user= etc has to come after the ?

I'm pretty sure 1.36 supports cameras with those type of streams that one is just mistyped..
User avatar
borozenetsww
Posts: 42
Joined: Tue Sep 12, 2023 1:55 am

Re: The memory fills progressively

Post by borozenetsww »

Work for me on last build.
rtsp://192.168.3.4:554/user=user_password=pass_channel=1_stream=0.sdp?real_stream
dougmccrary
Posts: 1322
Joined: Sat Aug 31, 2019 7:35 am
Location: San Diego

Re: The memory fills progressively

Post by dougmccrary »

Sorry Isaac. I meant that that string works in 1.36, but not 1.37. But then 1.37 system broke, so don't worry about it for now.
c128
Posts: 88
Joined: Sat Sep 14, 2013 3:36 pm

Re: The memory fills progressively

Post by c128 »

I'm seeing the same issue on a system that's basically remained the same (in terms of camera and configuration) for a couple of years.

I'm currently running 1.37.50~20240129103951-jammy, and I can't really tell with which release it started happening, but server memory is slowly eaten up, moving onto swap being exhausted until there's an eventual OOM auto kill of zmc.

You can actually watch it happening in real time in htop, it's quite quick:
htop rising emory
htop rising emory
htop_zm_rising_memory.gif (255.46 KiB) Viewed 3498 times
Nothing of note in the default log output.
User avatar
iconnor
Posts: 3197
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: The memory fills progressively

Post by iconnor »

Hey, are people who are experiencing this using Decoding=OnDemand?

I think I found the problem, and it's because Analysis is waiting for decode and so capture thread is just filling up the queue.

new package building should fix this.
c128
Posts: 88
Joined: Sat Sep 14, 2013 3:36 pm

Re: The memory fills progressively

Post by c128 »

iconnor wrote: Tue Jan 30, 2024 2:41 pm Hey, are people who are experiencing this using Decoding=OnDemand?

I think I found the problem, and it's because Analysis is waiting for decode and so capture thread is just filling up the queue.

new package building should fix this.
I am :) - thanks, will check out the new build when it's available.
marklg
Posts: 23
Joined: Sun Dec 17, 2023 1:43 am

Re: The memory fills progressively

Post by marklg »

iconnor wrote: Tue Jan 30, 2024 2:41 pm Hey, are people who are experiencing this using Decoding=OnDemand?

I think I found the problem, and it's because Analysis is waiting for decode and so capture thread is just filling up the queue.

new package building should fix this.
Mine are set to Decoding = Always or None. The ones growing biggest are all for my cameras that don't do ONVIF events correctly.

Regards,

Mark
c128
Posts: 88
Joined: Sat Sep 14, 2013 3:36 pm

Re: The memory fills progressively

Post by c128 »

Well, it seems to have worked for me.
I upgraded to 1.37.50~20240130093217-jammy a couple of hours ago, and memory usage has stayed pretty constant since.
dougmccrary
Posts: 1322
Joined: Sat Aug 31, 2019 7:35 am
Location: San Diego

Re: The memory fills progressively

Post by dougmccrary »

I don't know if you did something with this latest release, but I was able to get the source path on one line today.
The path in 1.36 is rtsp://192.168.1.218/user=admin_password=Buffy4@sdCA_channel=0_stream=1.sdp?real_stream

When I tried to paste it into 1.37, I got this:
Screenshot 2024-01-30 at 02-16-38 ZMX - Monitor - test2.png
Screenshot 2024-01-30 at 02-16-38 ZMX - Monitor - test2.png (163 KiB) Viewed 3368 times
But now it's all on one line, and works.
BTW, I see what you did with the start-up delays. Working nicely here now.
User avatar
iconnor
Posts: 3197
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: The memory fills progressively

Post by iconnor »

The @ symbol is confusing it. It is illegal in a url. Must be % encoded.
Post Reply