ZM crash every 10 seconds

[sagil]

I have 1 monitor defined
After running for several hours, ZM just crash and keeps crashing every 10 seconds
I'm running on Ubuntu 16
Output on 'good flow' :

Code: Select all

Jan 28 10:58:23 media-pc zmc_m1[19700]: INF [Monitor-1: 2662000 - Capturing at 20.00 fps]
Jan 28 10:59:04 media-pc zma_m1[17679]: INF [Monitor-1: 3076000 - Analysing at 20.00 fps]
Jan 28 10:59:13 media-pc zmc_m1[19700]: INF [Monitor-1: 2663000 - Capturing at 20.00 fps]
Jan 28 10:59:54 media-pc zma_m1[17679]: INF [Monitor-1: 3077000 - Analysing at 20.00 fps]
Jan 28 11:00:03 media-pc zmc_m1[19700]: INF [Monitor-1: 2664000 - Capturing at 20.00 fps]
Jan 28 11:00:43 media-pc zma_m1[17679]: INF [Monitor-1: 3078000 - Analysing at 20.41 fps]
Jan 28 11:00:52 media-pc zmc_m1[19700]: INF [Monitor-1: 2665000 - Capturing at 20.41 fps]
Jan 28 11:01:33 media-pc zma_m1[17679]: INF [Monitor-1: 3079000 - Analysing at 20.00 fps]
Jan 28 11:01:42 media-pc zmc_m1[19700]: INF [Monitor-1: 2666000 - Capturing at 20.00 fps]
Jan 28 11:01:53 media-pc zms[32119]: INF [Got signal 15 (Terminated), exiting]
Jan 28 11:01:53 media-pc zms[32119]: ERR [Unable to send stream frame: Broken pipe]
Jan 28 11:02:23 media-pc zma_m1[17679]: INF [Monitor-1: 3080000 - Analysing at 20.00 fps]
Jan 28 11:02:32 media-pc zmc_m1[19700]: INF [Monitor-1: 2667000 - Capturing at 20.00 fps]
Jan 28 11:03:13 media-pc zma_m1[17679]: INF [Monitor-1: 3081000 - Analysing at 20.00 fps]
Jan 28 11:03:22 media-pc zmc_m1[19700]: INF [Monitor-1: 2668000 - Capturing at 20.00 fps]
Jan 28 11:04:03 media-pc zma_m1[17679]: INF [Monitor-1: 3082000 - Analysing at 20.00 fps]
Jan 28 11:04:12 media-pc zmc_m1[19700]: INF [Monitor-1: 2669000 - Capturing at 20.00 fps]

Then this happens:

Code: Select all

Jan 28 14:42:30 media-pc zmc_m1[19700]: INF [Monitor-1: 2932000 - Capturing at 20.41 fps]
Jan 28 14:43:02 media-pc zmwatch[31487]: INF [Restarting capture daemon for Monitor-1, time since last capture 13 seconds (1517143382-1517143369)]
Jan 28 14:43:02 media-pc zmdc[31429]: INF ['zmc -m 1' sending stop to pid 19700 at 18/01/28 14:43:02]
Jan 28 14:43:02 media-pc zmc_m1[19700]: INF [Got signal 15 (Terminated), exiting]
Jan 28 14:43:05 media-pc zmdc[31429]: INF ['zmc -m 1' exited normally]
Jan 28 14:43:05 media-pc zmdc[31429]: INF [Starting pending process, zmc -m 1]

And it gets into this kind of loop : (it keeps crashing..)

Code: Select all

Jan 28 14:43:02 media-pc zmdc[31429]: INF ['zmc -m 1' sending stop to pid 19700 at 18/01/28 14:43:02]
Jan 28 14:43:05 media-pc zmdc[31429]: INF ['zmc -m 1' exited normally]
Jan 28 14:43:05 media-pc zmdc[31429]: INF [Starting pending process, zmc -m 1]
Jan 28 14:43:05 media-pc zmdc[31429]: INF ['zmc -m 1' starting at 18/01/28 14:43:05, pid = 22600]
Jan 28 14:43:05 media-pc zmdc[22600]: INF ['zmc -m 1' started at 18/01/28 14:43:05]
Jan 28 14:43:05 media-pc zmdc[31429]: ERR ['zmc -m 1' exited abnormally, exit status 255]
Jan 28 14:43:10 media-pc zmdc[31429]: INF [Starting pending process, zmc -m 1]
Jan 28 14:43:10 media-pc zmdc[31429]: INF ['zmc -m 1' starting at 18/01/28 14:43:10, pid = 22605]
Jan 28 14:43:10 media-pc zmdc[22605]: INF ['zmc -m 1' started at 18/01/28 14:43:10]
Jan 28 14:43:13 media-pc zmdc[31429]: INF ['zmc -m 1' crashed, signal 134]
Jan 28 14:43:15 media-pc zmdc[31429]: INF ['zmc -m 1' starting at 18/01/28 14:43:15, pid = 22622]
Jan 28 14:43:15 media-pc zmdc[22622]: INF ['zmc -m 1' started at 18/01/28 14:43:15]
Jan 28 14:43:16 media-pc zmdc[31429]: INF ['zmc -m 1' crashed, signal 134]
Jan 28 14:43:25 media-pc zmdc[31429]: INF ['zmc -m 1' starting at 18/01/28 14:43:25, pid = 22687]
Jan 28 14:43:25 media-pc zmdc[22687]: INF ['zmc -m 1' started at 18/01/28 14:43:25]
Jan 28 14:43:27 media-pc zmdc[31429]: INF ['zmc -m 1' crashed, signal 134]
Jan 28 14:43:36 media-pc zmdc[31429]: INF ['zmc -m 1' starting at 18/01/28 14:43:36, pid = 22695]
Jan 28 14:43:36 media-pc zmdc[22695]: INF ['zmc -m 1' started at 18/01/28 14:43:36]
Jan 28 14:43:36 media-pc zmdc[31429]: INF ['zmc -m 1' crashed, signal 134]
Jan 28 14:43:46 media-pc zmdc[31429]: INF ['zmc -m 1' starting at 18/01/28 14:43:46, pid = 22711]
Jan 28 14:43:46 media-pc zmdc[22711]: INF ['zmc -m 1' started at 18/01/28 14:43:46]
Jan 28 14:43:46 media-pc zmdc[31429]: INF ['zmc -m 1' crashed, signal 134]

After I restart ZN, everything works again

[rockedge]

is the camera causing this error? I had some cheap HOOTOO IP cams, a Chinese knock off of a FOSCAM, that would run after powering them up and ZM would stream them nicely. Then began the same type of zmc crashing and restarting...... I had to restart or re-save the cameras or power off and on the cameras and they would work again for a while. Then begin to crash and start like you are seeing. Short time later the HOOTOO's began to die completely and would not even have a web interface. Power light on but totally dead and un-responsive......point is the cameras while they were failing caused same set of errors and log reports.

perhaps check to be sure the camera(s) is functioning correctly for extended periods of time to be sure it is ZM at all.

[sagil]

is the camera causing this error? I had some cheap HOOTOO IP cams, a Chinese knock off of a FOSCAM, that would run after powering them up and ZM would stream them nicely. Then began the same type of zmc crashing and restarting...... I had to restart or re-save the cameras or power off and on the cameras and they would work again for a while. Then begin to crash and start like you are seeing. Short time later the HOOTOO's began to die completely and would not even have a web interface. Power light on but totally dead and un-responsive......point is the cameras while they were failing caused same set of errors and log reports.

perhaps check to be sure the camera(s) is functioning correctly for extended periods of time to be sure it is ZM at all.

thanks for replying !
The camera is an hacked xiaomi yi
But I don't think it's related since it only feeds ZM with rtsp

[Alec]

Jan 28 14:43:02 media-pc zmwatch[31487]: INF [Restarting capture daemon for Monitor-1, time since last capture 13 seconds (1517143382-1517143369)]

I believe that this means Zoneminder lost the rtsp stream 13 seconds ago . . . and then it starts trying to reconnect to the stream and it fails.

Try connecting to the camera stream some other way while this is going on and check if it is delivering a viewable stream.

[sagil]

Jan 28 14:43:02 media-pc zmwatch[31487]: INF [Restarting capture daemon for Monitor-1, time since last capture 13 seconds (1517143382-1517143369)]

I believe that this means Zoneminder lost the rtsp stream 13 seconds ago . . . and then it starts trying to reconnect to the stream and it fails.

Try connecting to the camera stream some other way while this is going on and check if it is delivering a viewable stream.

Well, the rtsp is pretty solid and I'm not sure this is the issue
I start thinking it has something to do with the 'modect' option and resources.
I've now changed the state to 'monitor' and Ill see if that makes any difference.

[zd59]

"time since last capture 13 seconds" - it's obvious, that camera failed, not ZM.

I had similar problems with my camera. Cameras are very sensible on stream capture settings.
I own two cameras and works perfectly with "source type = ffmpeg" the other one worked a few hours, then started to drop connections (trigger fired with blue screen instead of video). After camera reboot it works OK undetermined time, sometimes half a day, sometimes a few hours, then drops again.
There were a "smear" effect also sometimes not only drops of stream. Lasts months to solve this by long testings.
Try with different camera settings in ZM:
Source type: remote/ffmpeg
Remote Method: unicast/multicast...
and others

As of above, be convinced, the source of problem IS camera settings.

And to speed up your tests: Try to watch live video with VLC - leave it over night. If there were no drops, you are in the right direction.

And please report your observations here.

[sagil]

"time since last capture 13 seconds" - it's obvious, that camera failed, not ZM.

I had similar problems with my camera. Cameras are very sensible on stream capture settings.
I own two cameras and works perfectly with "source type = ffmpeg" the other one worked a few hours, then started to drop connections (trigger fired with blue screen instead of video). After camera reboot it works OK undetermined time, sometimes half a day, sometimes a few hours, then drops again.
There were a "smear" effect also sometimes not only drops of stream. Lasts months to solve this by long testings.
Try with different camera settings in ZM:
Source type: remote/ffmpeg
Remote Method: unicast/multicast...
and others

As of above, be convinced, the source of problem IS camera settings.

And to speed up your tests: Try to watch live video with VLC - leave it over night. If there were no drops, you are in the right direction.

And please report your observations here.

Thank you for your detailed answer.
I thing I'm not sure I understand, how changing zm settings affects the camera feed in VLC ?
Did I misunderstand something?

[zd59]

Sorry for an unclear advice regarding VLC. With the settings you used in ZM try to watch video with VLC or ffplay:
For instance:
ffplay.exe rtsp://admin:PASSWORD@Camera1_IP/0 -rtsp_transport tcp
and the other camera:
ffplay.exe "rtsp://admin:PASSWORD@Camera2_IP/cam/realmonitor?channel=1&subtype=0" -rtsp_transport tcp

As I tested, ffplay is best for that, VLC is unstable. If the capture stream works in ffplay, then sure will work in Zoneminder

[sagil]

Sorry for an unclear advice regarding VLC. With the settings you used in ZM try to watch video with VLC or ffplay:
For instance:
ffplay.exe rtsp://admin:PASSWORD@Camera1_IP/0 -rtsp_transport tcp
and the other camera:
ffplay.exe "rtsp://admin:PASSWORD@Camera2_IP/cam/realmonitor?channel=1&subtype=0" -rtsp_transport tcp

As I tested, ffplay is best for that, VLC is unstable. If the capture stream works in ffplay, then sure will work in Zoneminder

Ok then, I'll do some tests and report back as I'll have some conclusions.

Thank you!

[zd59]

I have ZM-1.30.0 on Debian and today:
uptime = 13:16:37 up 330 days, 18:05
Since I solved cameras issues this is "set and forget" system, rock solid. I only watch some recorded events occasionally.
The most "work" regardless the whole system is cleaning camera lenses off spider's webs.

[sagil]

I have ZM-1.30.0 on Debian and today:
uptime = 13:16:37 up 330 days, 18:05
Since I solved cameras issues this is "set and forget" system, rock solid. I only watch some recorded events occasionally.
The most "work" regardless the whole system is cleaning camera lenses off spider's webs.

Haha, I hope I'll get to the same state!

[zd59]

Don't worry, you came to the right place, to help you.
The other is bad (chip China ) camera as Rockedge (the respected developer) replied. There is no help for that.

[sagil]

1st update.
I've changed ZM mode to 'monitor' instead of 'modect'.
In addition, I've lowered the 'Maximum FPS' to 25 (from 35)
On buffers tab I've lowered 'Image Buffer Size' to 100
Warmup Frames to 25
Pre Event Image Count to 50
Post Event Image Count to 250
Stream Replay Image Buffer to 250

Here is the ffplay data from about 12 hours run:

Code: Select all

user@pc:~$ ffplay rtsp://1.1.1.1:554/ch0_0.h264
ffplay version 3.3.1-1~16.04.york0 Copyright (c) 2003-2017 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
  configuration: --prefix=/usr --extra-version='1~16.04.york0' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresamp
le --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enabl
e-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enabl
e-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --e
nable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
Input #0, rtsp, from 'rtsp://1.1.1.1:554/ch0_0.h264': f=0/0   
  Metadata:
    title           : H.264 Program Stream, streamed by the LIVE555 Media Server
    comment         : ch0_0.h264
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 20.08 fps, 20.08 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: aac (LC), 8000 Hz, stereo, fltp
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet   
[rtsp @ 0x7f2d7c000920] RTP: missed 723 packets
[h264 @ 0x7f2d7c15b240] Increasing reorder buffer to 1
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet   
[rtsp @ 0x7f2d7c000920] RTP: missed 65 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet2   
[rtsp @ 0x7f2d7c000920] RTP: missed 736 packets
[h264 @ 0x7f2d7c14e820] Increasing reorder buffer to 7
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet2   
[rtsp @ 0x7f2d7c000920] RTP: missed 145 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[h264 @ 0x7f2d7c14e820] left block unavailable for requested intra4x4 mode -1
[h264 @ 0x7f2d7c14e820] error while decoding MB 0 25, bytestream 5103
[h264 @ 0x7f2d7c14e820] concealing 1649 DC, 1649 AC, 1649 MV errors in I frame
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 4 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[h264 @ 0x7f2d7c14e820] error while decoding MB 60 31, bytestream -6
[h264 @ 0x7f2d7c14e820] concealing 1109 DC, 1109 AC, 1109 MV errors in P frame
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[h264 @ 0x7f2d7c14e820] concealing 209 DC, 209 AC, 209 MV errors in P frame
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[h264 @ 0x7f2d7c15b240] error while decoding MB 1 39, bytestream -6
[h264 @ 0x7f2d7c15b240] concealing 528 DC, 528 AC, 528 MV errors in P frame
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 2 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 208 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet3   
[rtsp @ 0x7f2d7c000920] RTP: missed 37 packets
[rtsp @ 0x7f2d7c000920] max delay reached. need to consume packet4   
[rtsp @ 0x7f2d7c000920] RTP: missed 1 packets
[h264 @ 0x7f2d7c14e820] left block unavailable for requested intra mode
[h264 @ 0x7f2d7c14e820] error while decoding MB 0 26, bytestream 23651
[h264 @ 0x7f2d7c14e820] concealing 1569 DC, 1569 AC, 1569 MV errors in I frame
47009.62 A-V:  0.235 fd=3343 aq=    0KB vq=    0KB sq=    0B f=2/4   4   

Now I've changed to 'modect' and started another ffplay run.

[zd59]

Hi sagil!

For a video surveillance it is enough to have < 10 FPS.
As you can see from ffplay log, Rockedge was right. Bad RTP stream, a lot of missed packets, quite a bit of >10 missed packets in sequence.
Try with lower FPS, except you have fast moving targets (car plates recognition). When errors vanish, increase FPS, until reappear, then select highest FPS without errors in RTP stream. Maybe you can select a different coding of a video in camera, disable motion detection in it e.t.c.
How is your camera connected? Sometimes WiFi cause trouble.
What is your CPU load on machine, error logs there..

As you can see, there are plenty of options to consider.

[sagil]

Hi sagil!

For a video surveillance it is enough to have < 10 FPS.
As you can see from ffplay log, Rockedge was right. Bad RTP stream, a lot of missed packets, quite a bit of >10 missed packets in sequence.
Try with lower FPS, except you have fast moving targets (car plates recognition). When errors vanish, increase FPS, until reappear, then select highest FPS without errors in RTP stream. Maybe you can select a different coding of a video in camera, disable motion detection in it e.t.c.
How is your camera connected? Sometimes WiFi cause trouble.
What is your CPU load on machine, error logs there..

As you can see, there are plenty of options to consider.

Hi !
As far as I've read, 'Maximum FPS' is not the FPS, moreover, this is just FPS in ZM, it won't affect the settings in ffplay stream.
ZM recommends changing the FPS through the camera settings so it won't throttle the CPU, but unfortunately it's not possible in my camera.

CPU usage is pretty high since I've turned on the camera (see attached image, I've turned it ON at 8:30 pm), however, there is no change since I've set it to Modect instead of Monitor.