Released 1.36.25 The Memory Remains

[iconnor]

# Changes since 1.36.24

- add build for ubuntu kinetic
- fix javascript error on zone edit
- fix deprecation error on php8 due to implicit conversion to integer when displaying event duration
- Update ZM_MIN_RTSP_PORT description
- fix some javascript errors during page transition
- Ignore errors when decoding log message
- add detection of out of order packets from ffmpeg
- Keep track of max_keyframe_interval and log it when complaining
- fix hang during logrotate due to waiting in packetqueue for decode
- Remove warning about maxImageBuffer. Will be handled better in queuePacket.
- Fix snapshot jpeg not being created early enough
- finally fix (we think) hung zmu/zms processes due to race in db thread creation.
- Update material icons to v1.11.10
- Add a button to event view to jump to this event time in montage review
- fix different button heights when using font awesome vs material icons
- Add a back to frames button from frame view
- Use HTTP_X_FORWARDED_HOST or HTTP_X_FORWARDED_SERVER if present to get correct hostname to use when behind a reverse proxy.
- Handle case where time_base is not set in the codec. Fixes h265 not playing through zms
- When there are less than 3 storage areas, just list them in the header instead of making it a dropdown
- fix problems with migrateHash

1.36.25
Full Changelog

[pat2]

working fine. thanks

[Heimo]

I have updatet to 1.36.25 and I got the problem, that logout is not working. By clicking on the username it shows the actual screen

Can anybody help?

[Heimo]

I have updatet to 1.36.25 and I got the problem, that logout is not working. By clicking on the username it shows the actual screen

Can anybody help?

it works via https

[keithp]

I think its working better- I'm still have cameras disconnecting every so often but after I upgraded on 9/3 I have been able to go back to Mocord. Analysis seem to be fine maybe a bit slow to assess alarms.

The problem I have now is event review and clean up. If I try to pull up all my events the system load goes over 100 and eventually the page will timeout and give me "SQLSTATE[HY000] [1040] Too many connections" messages. I have to log into the system and restart zoneminder and apache2 (I think it was just apache2 since the load was still pegged after restarting zoneminder). Since .23 the disk purge filter has not been working. I had extra space, so I've just been adding to the system to keep ahead of it. I thought it might be part of the recent issues but something is wrong.

Manually running the job yields no events unless I pull off the last part of the filter which is "and End Date/Time is not NULL"

I think I was able to delete 1000 events- about a years worth of data but the space drop was only 1 percent which doesn't feel right. I've been able to hold a year of data on a 4Tb system or less and now I'm up to an 8Tb system that is now at 77%. Any sort of filter operation causes the system load to climb again as database connections are used and I end up having to restart apache2. Maybe something is off with date and time queries?

I'm concerned that once that system fills up again, everything will lock up and I won't be able to clean things up.

[iconnor]

There is a known issue where if it takes too long to load the list of events the ajax call times out and tries again. Eventually there are too many requests in flight.

Master has a fix, and will likely land in 1.36.26.

[keithp]

There is a known issue where if it takes too long to load the list of events the ajax call times out and tries again. Eventually there are too many requests in flight.

Master has a fix, and will likely land in 1.36.26.

Ok thanks for that. Is there a reason why the PurgeWhenFull filter from .34.x would stop working? Is there another way I can clean up events so that my disk doesn't fill up. I don't seem to be making a dent and I'm now at 80%.

[keithp]

Here's a screenshot of one aspect of the event issue. I'm back on Mocord and typically each event should be 10 minutes (600 seconds). I've seen the system sometimes choke during analytics and as a result, segments will be smaller (resulting in missed footage actually) but generally I'll have a pretty complete record.

What is happening now seems to be completely random. I'm not getting the normal 6 events or so per hour and most of what I have does NOT have an end date/time. Its NULL... that now explains the purge filter failing but what would be causing this? The attached image was a display of events from the last two hours (run at around 17:35 local time). Note only 1 event got marked with and end time.

Screenshot_20220907_173810.png (117.49 KiB) Viewed 57902 times

[Magic919]

Sounds like something is crashing. I presume it’s zmc.

[iconnor]

As to filters, look in your filter logs. Turn on debug, let's look at the SQL that gets logged. If you are using storage areas, then you have to tell it which area to look at.

As to the events with no end, zmc is crashing. Look in your logs.

[keithp]

I figured zmc might be crashing and restarting (I had to kick the buffer setting up again) but the camera restarts within 2 minutes so the large gaps still do not make sense. It appears that the purge disk filter is deleting events randomly...

Screenshot 2022-09-09 00.44.48.png (271.57 KiB) Viewed 57752 times

For science, I went back to mode Record around 11:05. You'll see that events are still spotty. I looked at event 244357 (the one highlighted) and the data was there. For some reason that event ran about 15 minutes and as a side note, the timebar over the video disappears after about 9 minute mark. The next event logged didn't occur until about two hours later (even 244392). Also note, none of the shown events have ending date /timestamps. I went to the logs and found this...

Code: Select all

Sep  8 11:35:06 zm-nvr zmc_m5[70714]: INF [zmc_m5] [Front Door: 000 - Opened new event 244357, section start]
Sep  8 11:50:00 zm-nvr zmc_m5[70714]: INF [zmc_m5] [Front Door: 26895 - Closing event 244357, section end forced 1662652200 - 1662651305 = 895 >= 600]
Sep  8 11:51:35 zm-nvr zmc_m5[71021]: INF [zmc_m5] [Front Door: 000 - Opened new event 244361, section start]
Sep  8 12:10:00 zm-nvr zmc_m5[71021]: INF [zmc_m5] [Front Door: 33250 - Closing event 244361, section end forced 1662653400 - 1662652293 = 1107 >= 600]
Sep  8 12:10:39 zm-nvr zmfilter_1[65985]: INF [Deleting event 244361 from Monitor 5 StartDateTime:2022-09-08 11:51:33 from /var/cache/zoneminder/events/5/2022-09-08/244361]

So, there was an event started at 11:51:35 (244361) but the disk purge filter deleted it as well as many other event across all the cameras and I don't understand why since the only thing I did to the filter was lower the purge percentage to 55.

Screenshot 2022-09-09 01.05.05.png (134.6 KiB) Viewed 57752 times

My problem appears to be the same as this:
viewtopic.php?t=31686

and you mentioned there too turning on debugging there and restarting the filter. Can you tell me how to do that or point me to a reference?

[Magic919]

Your filter says to delete events where the end date/time is not null. That's why these events with no end date/time get deleted.

Go into Options, Logging and set at least debug level 3 to help Isaac.

[keithp]

Your filter says to delete events where the end date/time is not null. That's why these events with no end date/time get deleted.

Go into Options, Logging and set at least debug level 3 to help Isaac.

Ok. Thanks for that.

As for the event deletion. That makes sense but if that was the root cause then none of the events listed in the image would have been present. That filter rule has always been there and I presume that that is the default since events usually get an ending date / timestamp. Easy enough to switch to the start date / timestamp for now.

This also just happened too. All the cams went down from about 10 minutes and then recovered:

Code: Select all

Sep  9 11:30:11 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Got signal 11 (Segmentation fault), crashing]
Sep  9 11:30:12 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Signal address is 0xd1d, from 0x560b4e7dc448]
Sep  9 11:30:12 zm-nvr zmc_m7[87186]: INF [zmc_m7] [Yard Front: 25523 - Closing event 244748, section end forced 1662737400 - 1662736798 = 602 >= 600]
Sep  9 11:30:12 zm-nvr zmc_m6[87176]: INF [zmc_m6] [Porch: 25696 - Closing event 244749, section end forced 1662737400 - 1662736799 = 601 >= 600]
Sep  9 11:30:13 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 0: /usr/bin/zmc(+0xc4481) [0x560b4e811481]]
Sep  9 11:30:13 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fc2e1338420]]
Sep  9 11:30:14 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 2: /usr/bin/zmc(+0x8f448) [0x560b4e7dc448]]
Sep  9 11:30:14 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 3: /usr/bin/zmc(+0x727e5) [0x560b4e7bf7e5]]
Sep  9 11:30:14 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 4: /usr/bin/zmc(+0xe33e1) [0x560b4e8303e1]]
Sep  9 11:30:15 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 5: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7fc2def19de4]]
Sep  9 11:30:15 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fc2e132c609]]
Sep  9 11:30:16 zm-nvr zmc_m5[87196]: ERR [zmc_m5] [Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fc2dec06133]]
Sep  9 11:30:16 zm-nvr zmc_m5[87196]: INF [zmc_m5] [Backtrace complete, please execute the following command for more information: addr2line -e /usr/bin/zmc 0x560b4e811481 0x7fc2e1338420 0x560b4e7dc448 0x560b4e7bf7e5 0x560b4e8303e1 0x7fc2def19de4 0x7fc2e132c609 0x7fc2dec06133]

Executing the line yields...

Code: Select all

zm-nvr:~# addr2line -e /usr/bin/zmc 0x560b4e811481 0x7fc2e1338420 0x560b4e7dc448 0x560b4e7bf7e5 0x560b4e8303e1 0x7fc2def19de4 0x7fc2e132c609 0x7fc2dec06133
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0

Something else I just noticed on my cameras 6 and 7 that was in the logs as I was putting this post together:

Code: Select all

Sep  9 11:52:12 zm-nvr zmc_m6[87758]: ERR [zmc_m6] [Can't run query INSERT INTO `Frames` (`EventId`, `FrameId`, `Type`, `TimeStamp`, `Delta`, `Score`) VALUES #012( 244752, 8500, 'Bulk', from_unixtime( 1662738518 ), 564.32, 0 ): Cannot add or update a child row: a foreign key constraint fails]
Sep  9 11:52:18 zm-nvr zmc_m7[87884]: ERR [zmc_m7] [Can't run query INSERT INTO `Frames` (`EventId`, `FrameId`, `Type`, `TimeStamp`, `Delta`, `Score`) VALUES #012( 244753, 8200, 'Bulk', from_unixtime( 1662738510 ), 544.63, 0 ): Cannot add or update a child row: a foreign key constraint fails]

Funny thing is, I don't see any messages like this for camera 5.

I'll check the log in a bit for SQL statements.

[iconnor]

You have a filter deleting the events as they are created.

[keithp]

You have a filter deleting the events as they are created.

I turned it off for now.