Inconsistent clip lengths

[superdipak]

I have been using zoneminder for years and am currently on ZM 1.36.33 with six monitors running in mocord mode. All my cameras are identical, with each camera streaming in at 2592X1944 / 15 FPS. I have configured each monitor to save recordings in section lengths of 300 secs. While this works as expected for most of the times, I have observed that sometimes a large number of recordings with 0 (zero) length also get created. This seems to happen randomly, for a random duration of time after which normalcy is restored. However when the problem does happen, I get hundreds of zero sec clips. Not only that, I find that sometimes (though less frequently), clips extending upto 40 minutes also get greated. My ZM runs on an i7 PC with 32 GB RAM and looking at the resource usage on the box, I can see the utilization hardly ever exceeds 50% both in terms if CPU and memory.

I understand this may be related to buffer count settings. However I have tried several combinations but nothing seems to work.

Here are my settings at the moment:

ImageBufferCount = 4
MaxImageBufferCount = 50
WarmupCount = 25
PreEventCount = 2
PostEventCount = 200
MinSectionLength = 60

The problem was first seen on 1.36.33 and I can't remenber having seen this with 1.34.XX

Any insights would be appreciated. Please let me know if I can play around with any other configuration parameters to sort this issue out.

Regards

Dipak Jha

[dougmccrary]

I can't explain the long clips, but in my experience short ones are only caused by the camera signal dropping out - either to camera or network problems.
The settings you list have almost no effect in mocord, since it's recording all the time anyway.

[Magic919]

Make sure the capture daemons aren't crashing.

[superdipak]

Thank you for your responses. I have double, triple checked all my connections but can't find anything wrong. I have a network monitoring system in place whose 72-hour camera ping reports (from the ZM box) are all green. I have also checked syslogs & kernel logs and can't find any signs of crashing daemons.

However, I do see the following in ZM logs from time to time. Can this be triggering off the inconsistencies in clip length that I have reported?

Code: Select all

Can't run query INSERT INTO `Events` ( `MonitorId`, `StorageId`, `Name`, `StartDateTime`, `Width`, `Height`, `Cause`, `Notes`, `StateId`,
 `Orientation`, `Videoed`, `DefaultVideo`, `SaveJPEGs`, `Scheme` ) VALUES ( 10, 1, 'New Event', from_unixtime( 1692913892 ), 1920, 1080,
  'Continuous', '', 1, 1, 0, '', 0, 'Medium' ): Deadlock found when trying to get lock; try restarting transaction

[dougmccrary]

Is your system on a ssd? That can make a world of difference. Storage can be HDD.

[superdipak]

Yes, the OS and ZM are on a SSD. However, ZM storage is on a 4TB HDD.

[Magic919]

The standard advice would be to tune the DB.

I’d suggest checking the capture processes, rather than just looking for messages they have stopped.

[burger]

Whenever I see clip lengths getting truncated it's because of lack of resources.

You are trying to mocord 6 2.5K cameras on an i7? How many cores? And why an i7 as opposed to a server CPU? You need like 16 or 24 actual cores, a server motherboard and CPU. Alternatively you can just use record & passthrough and then setup either linked monitors or have separate low res streams that do the mocord. Mocord on a 2K or 4K stream is CPU hungry due to how ZM works. It has been discussed before in the forum. Somewhere there is a recommendation on how many cores to use depending on stream res.

See: viewtopic.php?t=29953 and: viewtopic.php?f=14&t=30973

Get something like a Dell R720, they are cheap (<$300) and have 16 cores. I run one with about 40 cameras. Or you can go newer gen if your budget allows. If you are upgrading from 1.32 or 1.34, you might want to consider just starting a fresh 1.36 install. Somewhere in 1.34 there were the buffer changes that broke a lot of things. It's not worth trying to figure out the settings, just do a new install. If you want to import the old camera paths, see https://wiki.zoneminder.com/MySQL

[superdipak]

I made a couple of changes to the configuration to work around this problem:

1. Reduced source frame interval from 20 to 5 and FPS from 15 to 5 for all six of my cameras
2. Increased WATCH_CHECK_INTERVAL from 15 to 60 in ZM
3. Increased WATCH_MAX_DELAY from 45 to 180 in ZM

I know the last 2 changes above may be masking my underlying problem but with the above changes, the number of occurrences has come down massively from 100s to just 2 or 3 in a 24-hr period.

Having dug deeper into one of the occurrences of my problem, I can now see the following in ZM logs (database)

Code: Select all

|FROM_UNIXTIME(TimeKey) |component|file                |code|message                                                             |
|-----------------------|---------|--------------------|----|--------------------------------------------------------------------|
|2023-08-29 09:58:05.817|zmc_m6   |zm_ffmpeg_camera.cpp|INF |Unable to read packet from stream 0: error -541478725 "End of file".|
|2023-08-29 09:58:05.830|zmc_m6   |zmc.cpp             |ERR |Failed to capture image from monitor 2 CAM 2 (1/1)                  |
|2023-08-29 09:58:05.830|zmc_m6   |zm_monitor.cpp      |INF |CAM 2: image_count:1118551 - Closing event 3166, shutting down      |
|2023-08-29 09:58:08.052|zmc_m6   |zm_monitor.cpp      |INF |CAM 2: 1118530 - Opened new event 3167, section start               |

Looks like zmc is unable to read a packet from the stream of the affected camera and is therefore truncating the section.

Are there any workarounds for this kind of a situation?

[superdipak]

Please, can anyone provide any insights into this issue? I still get a few prematurely terminated clips every day and in every single case, I see the following in the logs:

Code: Select all

Unable to read packet from stream 0: error -541478725 "End of file".

[dougmccrary]

Does it happen on all cameras? I don't think you've said if they're wifi or wired. You did say they are identical.

[superdipak]

I have six identical cameras connected to the ZM box over wired (POE) circuits. The cameras are continuously monitored by a software called nagios running on the same host as zoneminder. Nagios monitoring reports have not shown a single missed ping for months. However, I still keep getting these ZM error messges and inconsistent clip lengths for all my cameras.

[dougmccrary]

OK. Have you looked at the load into the server, byte-wise? Is it anywhere near the lan capacity?

[iconnor]

Some cameras do this to prevent timestamp wrap around.... but more generally I see them when zmatch kills zmc because it hasn't captured anything in a while. Is there anything in your zmwatch logs about that?

[superdipak]

I have checked the network bandwidth usage on the server and that does not appear to be a problem. Please see attached a screenshot of the output of nload with all 6 cameras streaming. I am not an expert in this area but for a 10/100/1000 gigabit NIC card that the machine has, the network load does not appear to be the bottleneck at all.

I have also checked zmwatch logs and can confirm there is nothing odd there.

@iconnor: please can you elaborate on the "timestamp wrap around" you have mentioned in your post? How can I figure out if my cameras are attempting to prevent a timestamp wrap around? Is there a workaround?