Getting Many Issues upgrading from 1.33.8 to 1.34.6 or 1.35.1

[seven]

I've been holding off upgrades for some time now due to these issues.

Upgrading from 1.33.8~20190510143016 to either
1.34.6
or
1.35.1~20200310184748
from iconnor ppa for xenial

Appears to only occur on one Cam, both are Onif IP cam's but cam 2 appears to crash, while cam 3 appears to be fine... (there is no cam 1),

Code: Select all

2020-03-20 22:06:34.356174 zmc_m3[20992].WAR-zm_ffmpeg_camera.cpp/951 [ImageBufferCount 20 is too small. Needs to be at least 31. Either increase it or decrease time between keyframes]
2020-03-20 22:06:34.300550 zmdc[20943].INF-zmdc.pl ['zmc -m 2' starting at 20/03/20 22:06:34, pid = 24113]
2020-03-20 22:06:34.300550 zmdc[24113].INF-zmdc.pl ['zmc -m 2' started at 20/03/20 22:06:34]
2020-03-20 22:06:34.253980 zmdc[20943].INF-zmdc.pl [Starting pending process, zmc -m 2]
2020-03-20 22:06:33.640004 zmc_m3[20992].INF-zm_monitor.cpp/2546 [House Rear: images:7100 - Capturing at 14.29 fps, capturing bandwidth 63936bytes/sec]
2020-03-20 22:06:33.223570 zmdc[20943].INF-zmdc.pl ['zmc -m 2' crashed, signal 8]
2020-03-20 22:06:33.164800 zmdc[20943].WAR-zmdc.pl ['zmc -m 2' has not stopped at 20/03/20 22:06:33 after 60 seconds. Sending KILL to pid 23787]
2020-03-20 22:06:32.366506 zmc_m3[20992].WAR-zm_ffmpeg_camera.cpp/951 [ImageBufferCount 20 is too small. Needs to be at least 31. Either increase it or decrease time between keyframes]
2020-03-20 22:06:30.356112 zmc_m3[20992].WAR-zm_ffmpeg_camera.cpp/951 [ImageBufferCount 20 is too small. Needs to be at least 31. Either increase it or decrease time between keyframes]
2020-03-20 22:06:29.383420 zma_m3[20999].INF-zm_monitor.cpp/1297 [House Rear: 6400 - Analysing at 14.29 fps]
2020-03-20 22:06:28.356137 zmc_m3[20992].WAR-zm_ffmpeg_camera.cpp/951 [ImageBufferCount 20 is too small. Needs to be at least 31. Either increase it or decrease time between keyframes]
2020-03-20 22:06:26.959560 zmc_m3[20992].INF-zm_monitor.cpp/2546 [House Rear: images:7000 - Capturing at 16.67 fps, capturing bandwidth 113930bytes/sec]
2020-03-20 22:06:26.902540 zmdc[24095].INF-zmdc.pl ['zma -m 2' started at 20/03/20 22:06:26]
2020-03-20 22:06:26.901190 zmdc[20943].INF-zmdc.pl ['zma -m 2' starting at 20/03/20 22:06:26, pid = 24095]
2020-03-20 22:06:26.710290 zmdc[20943].INF-zmdc.pl ['zmc -m 2' sending stop to pid 23787 at 20/03/20 22:06:26]
2020-03-20 22:06:26.569310 zmdc[20943].INF-zmdc.pl ['zma -m 2' exited, signal 14]
2020-03-20 22:06:26.538770 zmdc[20943].INF-zmdc.pl ['zma -m 2' sending stop to pid 24044 at 20/03/20 22:06:26]

[seven]

I get similar logs in 1.33.8 but camera is online and functioning, camera is offline in the other versions or constantly alarming with errors "Reacquiring Signal"

Code: Select all

2020-03-20 22:29:50.414694 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:50.360573 zma_m3[25432].INF-zm_monitor.cpp/1269 [House Rear: 6000 - Analysing at 12.50 fps]
2020-03-20 22:29:50.329562 zmc_m3[25616].WAR-zm_ffmpeg.cpp/64 [Invalid UE golomb code
]
2020-03-20 22:29:50.295707 zmc_m3[25427].WAR-zm_ffmpeg_camera.cpp/884 [ImageBufferCount 20 is too small.  Needs to be at least 61. Either increase it or decrease time between keyframes]
2020-03-20 22:29:50.262115 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Invalid UE golomb code
]
2020-03-20 22:29:50.228197 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:50.114280 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:49.986365 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:49.867239 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:49.764549 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:49.627819 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:49.514700 zmc_m3[25427].WAR-zm_ffmpeg.cpp/64 [Error parsing AU headers
]
2020-03-20 22:29:49.467572 zma_m2[25423].INF-zm_monitor.cpp/1635 [House Front: 4411 - Gone into prealarm state]
2020-03-20 22:29:49.441495 zma_m2[25423].WAR-zm_monitor.cpp/1392 [Signal: Reacquired]

[iconnor]

Please enable debug logs

[seven]

OK I think I've enabled debug logs but can see anything that sheds any light to me anyway, perhaps these are ffmpeg errors causing issues with Monitor 2.....

Code: Select all

2020-03-25 20:53:00.191177 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:53:00.163417 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:53:00.112870 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:53:00.087741 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [mmco: unref short failure]
2020-03-25 20:53:00.055898 zmc_m3[10369].WAR-zm_ffmpeg.cpp/70 [Error parsing AU headers]
2020-03-25 20:53:00.020042 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:52:59.988842 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.953186 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [mmco: unref short failure]
2020-03-25 20:52:59.919134 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.885394 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:52:59.853892 zmc_m3[10369].WAR-zm_ffmpeg.cpp/70 [Error parsing AU headers]
2020-03-25 20:52:59.818445 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [mmco: unref short failure]
2020-03-25 20:52:59.784403 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.750614 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:52:59.719176 zmc_m3[10369].WAR-zm_ffmpeg.cpp/70 [Error parsing AU headers]
2020-03-25 20:52:59.683550 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [mmco: unref short failure]
2020-03-25 20:52:59.652134 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.615920 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:52:59.582378 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.548887 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [mmco: unref short failure]
2020-03-25 20:52:59.517008 zmc_m3[10369].WAR-zm_ffmpeg.cpp/70 [Error parsing AU headers]
2020-03-25 20:52:59.471891 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:52:59.448674 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.422868 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [mmco: unref short failure]
2020-03-25 20:52:59.406456 zma_m2[10622].DB1-zm_monitor.cpp/531 [Waiting for capture daemon last_write_index(101), last_write_time(0)]
2020-03-25 20:52:59.387562 zmc_m3[10369].DB1-zm_ffmpeg_camera.cpp/981 [raw frame from decoder: format 12 yuvj420p 1920x1080 linesize:1920x960 pts: -9223372036854775808]
2020-03-25 20:52:59.167358 zmc_m2[10527].WAR-zm_ffmpeg.cpp/70 [illegal short term buffer state detected]
2020-03-25 20:52:59.135846 zmc_m3[10369].WAR-zm_ffmpeg.cpp/70 [Error parsing AU headers]

[seven]

some time ago I was running on Master releases due to some features I was after, but have not been able to upgrade since 1.33.8~20190510143016-xenial, first failed Update was 1.33.15~20191213181247-xenial & 1.33.16~20200113170925-xenial

Now on 1.35.1~20200310184748-xenial but seeing the same issues in 1.34.6-xenial1

[iconnor]

I havn't seen those errors before.

Do you have values in your MaxFPS or AlarmMAXFPS fields? If so, please clear them.

[seven]

Nope Neither Monitor has those settings set.

[seven]

Interestingly I see similar logs in 1.33.8 so perhaps not related

I'm not entirely sure I enabled debug logging correctly is there some guide?

[iconnor]

Your debug is fine. I just don't know what the problem is. ffmpeg is complaining about the stream. How are these cameras connected, ethernet or wifi?

[seven]

Ethernet

[bbunge]

Have you tried your system on Bionic?

[seven]

Have you tried your system on Bionic?

Why would that work? does zoneminder use the system installed ffmpeg?
or its own internally forked version from upstream code?

[iconnor]

We use the system ffmpeg libraries.

[seven]

No worries, I'll might try running the upgrade in a docker container then.

[seven]

Still Seeing similar issues in docker zm 1.34.6 using the 18.04 (very similar to what was seen in xenial)

I've also tried replacing /usr/bin/ffmpeg with the snapd ffmpeg package and no change in 1.33.8 I'll try again in 1.34.6 with the snapd ffmpeg

Code: Select all

/usr/bin/ffmpeg -version
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100

verses snapd ffmpeg

Code: Select all

/snap/bin/ffmpeg -version
ffmpeg version n4.1.4 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
configuration: --prefix= --prefix=/usr --disable-debug --disable-doc --disable-static --enable-avisynth --enable-cuda --enable-cuvid --enable-libdrm --enable-ffplay --enable-gnutls --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopus --enable-libpulse --enable-sdl2 --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-nonfree --enable-nvenc --enable-omx --enable-openal --enable-opencl --enable-runtime-cpudetect --enable-shared --enable-vaapi --enable-vdpau --enable-version3 --enable-xlib
libavutil      56. 22.100 / 56. 22.100
libavcodec     58. 35.100 / 58. 35.100
libavformat    58. 20.100 / 58. 20.100
libavdevice    58.  5.100 / 58.  5.100
libavfilter     7. 40.101 /  7. 40.101
libswscale      5.  3.100 /  5.  3.100
libswresample   3.  3.100 /  3.  3.100
libpostproc    55.  3.100 / 55.  3.100