zm_videostore.cpp warning on 1.36.18, was also present on 1.36.17

[Andyrh]

Ok, I will experiment for a while with camera and ZM settings and see if hit the magic fix. With a cycle time of about an hour it will take a while.

[Andyrh]

Update:
I changed reorder_queue_size to 400 and now the events are about 25 hours apart.
I have moved on to changing things on the camera.

[iconnor]

Yeah I havn't found a setting that fixes it. I also have identical cameras with identical firmware settings etc and one does it and the other doesn't.

I think I'm going to have to figure out how to best deal with it in zm...

Isaac

[Andyrh]

I have so far made the error go away. On the camera I disabled the audio portion of the stream. There is no mic on this camera.
With audio on and reorder_queue_size=400 the errors occur about ever 25 hours. Without setting reorder_queue_size the errors occur every 30 - 60 minutes.
With audio off at the source and reorder_queue_size not set the errors have not repeated in 36 hours. I tested by enabling the audio on the camera and the errors returned to the 30-60 minute schedule.

One oddity, when I make a change to the camera settings a set of 6 of the errors occur, but they have not repeated. Monitor setting changes do not produce the errors.
 
This particular camera is a IP4M-1028EW running software version 2.420.AC01.3.R, Build Date: 2018-02-06. That is the newest version.
The hires feed did not have audio enable and has not generated the errors.

I hope the details help someone.

[iconnor]

Darn... the cameras I have that do this do not have audio turned on.

[Andyrh]

Now that most of the noise is cleared, I have 3 cameras that are generating the messages. They all did it at the weekly restart.

If I run across anything new, I will post it.

Code: Select all

11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	pkt.dts(0) must be <= pkt.pts(-18000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	non increasing dts, fixing. our dts -18000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	pkt.dts(0) must be <= pkt.pts(-36000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	non increasing dts, fixing. our dts -36000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	pkt.dts(0) must be <= pkt.pts(-54000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	non increasing dts, fixing. our dts -54000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	pkt.dts(0) must be <= pkt.pts(-72000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	non increasing dts, fixing. our dts -72000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	pkt.dts(0) must be <= pkt.pts(-90000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m7		3761445	WAR	non increasing dts, fixing. our dts -90000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	pkt.dts(0) must be <= pkt.pts(-18000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	non increasing dts, fixing. our dts -18000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	pkt.dts(0) must be <= pkt.pts(-36000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	non increasing dts, fixing. our dts -36000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	pkt.dts(0) must be <= pkt.pts(-54000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	non increasing dts, fixing. our dts -54000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	pkt.dts(0) must be <= pkt.pts(-72000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	non increasing dts, fixing. our dts -72000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	pkt.dts(0) must be <= pkt.pts(-90000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m8		3761444	WAR	non increasing dts, fixing. our dts -90000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	pkt.dts(0) must be <= pkt.pts(-18000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	non increasing dts, fixing. our dts -18000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	pkt.dts(0) must be <= pkt.pts(-36000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	non increasing dts, fixing. our dts -36000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	pkt.dts(0) must be <= pkt.pts(-54000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	non increasing dts, fixing. our dts -54000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	pkt.dts(0) must be <= pkt.pts(-72000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	non increasing dts, fixing. our dts -72000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	pkt.dts(0) must be <= pkt.pts(-90000).Decompression must happen before presentation.	zm_videostore.cpp	1409
11/21/23, 12:18:21 AM CST	zmc_m9		3761439	WAR	non increasing dts, fixing. our dts -90000 stream 0 last_dts 0. reorder_queue_size=0	zm_videostore.cpp	1398

[Andyrh]

I have found a work around.
I went on the assumption that the frames may be arriving too fast or the camera is not keeping up and making a mess of the frames. I changed the network ports on the 3 affected camera to 10/half with reorder_queue_size=40. After making the changes to all three cameras I am now no longer able to reproduce the problem. I was able to reproduce the issue by changing a setting on the cameras.

I do not know if this will help find the issue or suggest a better fix.

For those worried about the lower speed being a problem check the math, 10Mb/s should be plenty for many cameras. My 8MP camera is sending 3 streams, 2 to ZM and 1 to the Amcrest app. (1 hi-res, 2 low-res) This totals about 950kB/s (numbers from ZM) or 7.6Mb/s leaving some head room.

[iconnor]

I may have found a fix. I was incorrectly determining keyframe interval using the audio stream if present. So it should guess the keyframe interval better now...

[mikb]

I may have found a fix. I was incorrectly determining keyframe interval using the audio stream if present. So it should guess the keyframe interval better now...

I wonder if this is symptomatic of a camera maintaining the same DTS for a few frames in a row to try and bring the audio and pictures back into sync.

Under an old Windows AVI capture programme, which was written WAY more competently than the "Demo Win Cap" and derivatives, which assumed that exactly 30 fps (e.g.) occurred for every second, and that exactly 44,100 (e.g.) samples occurred on the audio.

This software handled the fact that REALITY is not perfect (VHS video source may not be exactly 30 fps, your audio hardware may not run at EXACTLY 44100 ...) by duplicating or dropping video frames to maintain sync.

It warned when it did this, so I would often see 24-25 mins of good behaviour, then a single "Drops = 1" because enough discrepancy had built up to require a resync. Perfect!

Now, if a camera is producing audio and video -- the same divergence in sync can happen over long periods of time. So, a strategy would be to insert or remove 1 video frame to resync. This means either you could get a DTS that duplicates two (or more?) frames running while it tries to resync.

Of course, FFMPEG tends to whinge about "non monotonic timestamps" which is just a fancy way of saying it's expected to always increase, and it didn't. I see these complaints from trying to FFMPEG process output from a GoPro knockoff, and think it's the same cause -- a group of non-monotonic (duplicated) timestamps every so many minutes as it gets the streams back in sync.

Wonder if that's why it is more a problem for cameras with audio? Just thinking out loud here

[Legion4]

Here is my experience in case it helps someone else:
I have one camera that regularly gets the "non increasing dts, fixing. our dts..." warning message. Adding reorder_queue_size=1001 made it less common, but turning off audio on the camera completely removed it. This is a good solution for me.

The camera is: LNZ3522
ZM version is: v1.36.35

--L4

[mikb]

but turning off audio on the camera completely removed it.

That does make me think that the camera is issuing video frames with identical timestamps every so often to keep the video and audio in sync, because it has a bit-more-audio than video building up. If it was the other way round, it could drop a video frame and you'd be none the wiser. To stop things getting out of sync, it does e.g.

V DTS 0: + Packet of audio
V DTS 1: + Packet of audio
.... time passes ...
V DTS 3344: + Packet of audio
V DTS 3344: + I seem to have one frame's worth of audio spare, hmmm ...
V DTS 3345: + Packet of audio ...

etc.

which upsets FFPMEG and similar "strict" video handling because time cannot stand still or go backwards. Each frame must be dated LATER than the previous one!

Turning off audio means it will never have to deal with that imbalance, just issues ever increasing timestamps.

It would be nice if FFMPEG had a "yes, I know, just deal with it" option, or a "WARN: not ERROR:" option.

[iconnor]

FFMPEG Used to allow duplicated timestamps but it got removed in 5.0 I think. Or at least I can't figure out how to turn it on anymore. Has been a huge PIA actually.

[eggenbef]

Here is my experience in case it helps someone else:
I have one camera that regularly gets the "non increasing dts, fixing. our dts..." warning message. Adding reorder_queue_size=1001 made it less common, but turning off audio on the camera completely removed it. This is a good solution for me.

The camera is: LNZ3522
ZM version is: v1.36.35

--L4

thank you - just double-checked the settings of the RLC-510A and P340 i am using. both cameras have audio disabled and the warnings only appear with version v1.36.35. no issue with 1.36.33. see here for details: viewtopic.php?t=33692. appreciate any further suggestions on how to get this issue resolved.

[iconnor]

How stable is the timestamp from the cameras? Are they using ntp?

You can send me a debug level 3 log from one of them and maybe I can figure it out.

[domlin]

Hi - just registered to post in case this is of any use.

I'd been seeing the 'non increasing dts' warnings, and also seeing frame order issues in my recorded events - I tried everything including disabling the audio stream and setting the 'reorder_queue_size' to various values, but nothing had any effect.

Finally out of desperation I changed network method from TCP to UDP ('Method' under 'Source' in the Monitor settings) and the issue was instantly fixed for all 3 of my monitors/cameras. Not sure if this is unique to my setup but thought i'd share in case it can help anyone else.

Thanks