ZoneMinder Forums

My first post(!)

I have trouble with ZM - after some hours (stretching randomly from a few hours up to more than a whole day) of successful running some cameras start to show a 100% grey image. Hitting the Save button in ZM's settings for that monitor (without making any changes) refreshes the image for some additional hours.
I'm a total beginner with ZM and have the following setup:
- Raspberry PI 2 (4 core) running Debian "Jessie" (latest NOOB)
- External disk for storing of ZM-data (MySQL/images/events)
- ZoneMinder 1.29 built from source (http://zoneminder.readthedocs.org/en/la ... ebian.html)

After some initial struggle with getting ZM and Apache to "marry" (enable site, enable CGI, location of cgi-bin folder, etc.) ZM finally started up!
I've added three cameras (all running latest available firmware):
2 x ACTi D72a (semi-professional outdoor 3MP fixed dome)
1 x AZTech WIPC408HD ("home use" Pan-Tilt thingy)

The AZTech seem to only provide a 1280x720 image while the ACTi can be configured with two independent streams from 640x360 up to 1920x1080 with various settings (H.264, MPEG, etc.).

The trouble is that the capturing from the ACTi cameras (I'm so far using 640x360, 3 fps to preserve CPU in the Raspberry) turn 100% grey after some hours, up to days.

I have experimented some with the bitrates (variable -> fixed) and H.264 settings (Baseline -> Main, but not yet High) to no avail.

When the problem has struck, I still see same amount of network traffic (e.g. with nethogs) and CPU load (top) dissipated by the associated zmc process. By opening the monitor settings for that camera (where RTSP, IP address, etc. details are set) and pressing Save (with no change!) a new period of successful capturing is "bought".

When viewing the cameras, the ACTi image sometimes (rather seldom) briefly flicker to 100% grey and quickly back to normal. I have not managed to find out if this glitch is fed from the camera, or "invented" in the zmc capturing or later.

I do not have any issues with the cheaper "home-use" camera running higher resolution and framerate.

Removing the AZTech monitor does not help (CPU load is reduced by some 30% down to 40%, but 70% is far from the 400% available from the four core processor of the RPi2). No glitches in the "dmesg" output or other logs in the Raspberry (/var/log/daemon,syslog,etc).

The problem often strikes one camera first and the other some time after - the image attached above show one monitor down.

All cameras are wired (no wireless) - no issues with network seen (so far...).

Here is a sample of warnings and errors I get from xmc (zm_remote_camera_rtsp.cpp) capturing one of the ACTi cameras:
Error while decoding frame 318728
Discarding incomplete frame 476810, 0 bytes
Discarding partial frame 476810, 56644 bytes
Packet in sequence, gap 14
Unexpected channel selector 239 in RTSP interleaved data
Unexpected format RTSP interleaved data, resyncing by 9 bytes
Unexpected format RTSP interleaved data, dumping 52 bytes
There are also some errors just containing a hex dump of varying bytes.


I have not been able to connect a "greyout" event to any of these warnings and errors, capture seem to continue even though they are output.

I'd appreciate any help I can get.

- Switch from H.264 to JPEG?
- Adjusting resolution
- Adjusting H.264/JPEG settings in camera?
- Switch from RTSP protocol to another?
- Any events (there are plenty of events recorded in the ZM logs - but I can not tell if they are relevant or not) to examine in more detail?
- Any additional logging I can enable to provide reasons for why zmc start to capture 100% grey after some time?
- Track same cam feed with another device, e.g. VLC Player to see if it also quit when ZM does?

Regards
/f

Try changing one of the monitors to ffmpeg instead of remote. Doesn't solve the problem, but does help identify it something environment related vs type of monitor you are using.

SteveGilvarry wrote:Try changing one of the monitors to ffmpeg instead of remote. Doesn't solve the problem, but does help identify it something environment related vs type of monitor you are using.

OK, done.

I changed the first monitor from "Remote" to ffmpeg and set its Source Path to rtsp://192.168.1.131:554/track2 and Remote Method to RTP/RTSP, no changes in camera.

Live feed on 1 & 2 had broken during the day, but picked up (for monitor 1 when swapping to ffmpeg and for monitor 2 when I used the "Save" trick)
Can you elaborate what the difference with this setup is?

What shall I be on the lookout for? Logs? Traces?

OK, it took a little bit less than two hours running number one of the "grey out" monitors on ffmpeg and we have got ourselves a new scenario.

xmc exited at 21:41:38 with an error status of 255 and was restarted.
The monitor of this camera is now constantly flickering between image / grey 100% / image / grey 100% / ... over and over again.
CPU load and network load look like before (even no difference from when running as Remote) - monitor 2 still on Remote show good image.

EDIT: Monitor 2 is now (after another 3 hours at 100% steady grey again as before) - monitor 1's zmc_m1 has restarted second time and the feed is still flickering grey/image/grey/image/...

Here are some events from the log:

2016-03-16 21:41:38.478702 zmc_m1 5953 INF Priming capture from rtsp://192.168.1.131:554/track2 zm_ffmpeg_camera.cpp 104
2016-03-16 21:41:38.475262 zmc_m1 5953 INF Starting Capture version 1.29.0 zmc.cpp 250
...
2016-03-16 21:41:38.254300 zmdc 5953 INF 'zmc -m 1' started at 16/03/16 21:41:38 zmdc.pl
2016-03-16 21:41:38.250260 zmdc 18602 INF 'zmc -m 1' starting at 16/03/16 21:41:38, pid = 5953 zmdc.pl
2016-03-16 21:41:38.237240 zmdc 18602 INF Starting pending process, zmc -m 1 zmdc.pl
2016-03-16 21:41:38.231800 zmdc 18602 ERR 'zmc -m 1' exited abnormally, exit status 255 zmdc.pl
...
2016-03-16 21:41:38.203150 zmc_m1 31484 FAT Unable to decode frame at frame 21634 zm_ffmpeg_camera.cpp 178
...
2016-03-16 19:44:50.283612 zmc_m1 31484 INF Priming capture from rtsp://192.168.1.131:554/track2 zm_ffmpeg_camera.cpp 104
2016-03-16 19:44:50.279843 zmc_m1 31484 INF Starting Capture version 1.29.0 mc.cpp 250


From first start of zmc_m1 on the suggested new "ffmpeg" setting to first crash of zmc_m1 there were no error outputs from zmc_m1, but the zmc_m1 crash were preceded (by less than 1/2 second) with some errors on xmc_m2 (still running as Remote) - can this zmc_m2 error be related to the problem?

2016-03-16 21:41:38.231800 zmdc 18602 ERR 'zmc -m 1' exited abnormally, exit status 255 zmdc.pl
2016-03-16 21:41:38.203150 zmc_m1 31484 FAT Unable to decode frame at frame 21634 zm_ffmpeg_camera.cpp 178
2016-03-16 21:41:37.988617 zmc_m2 31569 ERR 256: 00 00 01 25 b8 00 00 40 08 ff e1 dc 47 8b 37 62 33 2b 87 67 10 ed 43 03 10 9a 30 93 23 e1 ae 44 b6 02 98 24 64 7a e1 4d aa da fa 3a fc 82 8e 36 34 a5 0a 6c 8d c4 dc ed cc e1 c3 5c ce 65 c9 03 6a ff f9 fb d8 7f 57 df 5a ae c9 8a fd 40 12 3c d6 f1 36 4c 17 ec 29 eb a5 57 cd 9b c7 ea 76 ec c2 20 6a 0c 1f 1b b4 6a 2b 3e f3 05 ae 62 f2 ba a4 9e 03 79 5c 79 0b 6c d3 8e f8 b6 d4 89 be 7d 76 79 1d da 60 82 dd af 54 5b 9c 71 50 01 c2 b0 33 7c 5f b9 3f 2d 4c 9c 56 a5 c0 6b fd a4 52 91 20 4e 01 c7 d2 7e 5f 5d 34 0f 1f 37 7b 7f 2e 18 e7 34 06 31 75 2a cf ee 7c 00 e2 a9 27 7d c1 87 9f df b3 f3 35 fa 68 58 a0 68 8d 1d 2b fd 8b ef 8f 53 a9 cc 4a 52 11 3b a4 1e 69 b4 3f 7c 49 b7 85 6f 79 cb 56 0d 6f c2 e3 1b ba d1 41 34 27 94 90 89 42 fa 06 69 58 d4 5c 11 ea 49 58 77 89 2e zm_remote_camera_rtsp.cpp 318
2016-03-16 21:41:37.983841 zmc_m2 31569 ERR Error while decoding frame 21445 zm_remote_camera_rtsp.cpp 317
2016-03-16 21:41:37.678177 zmc_m2 31569 ERR 256: 00 00 01 21 e0 00 42 11 ff f1 95 30 4b 06 e3 df 1b 56 8b 07 32 5e 82 7f 06 f1 4d 3a 42 fb ca fd 4d ec e9 51 14 68 c1 a9 52 36 39 46 ad a5 5f 4b 72 b6 bf 3e c0 45 01 ab dc 67 cb e9 20 8e 28 b8 48 39 ad 56 aa 4b f9 32 e3 74 3c 0d a0 ff 4a ac 9c 66 74 6a 00 86 81 39 eb 01 bf 85 28 be cc 58 77 9e 3d 1c c1 48 aa e4 49 f0 3c 19 65 cf c7 d4 c7 77 5e ce db aa d3 10 71 ff 54 04 96 ae 43 17 d4 37 34 db 10 f8 7e 6f 62 06 1b 4c 17 e6 24 12 02 2d 54 17 45 0f c4 41 a3 33 da 3a 79 e2 79 e1 84 9b 7f 85 fb 6a 61 06 0c 03 4c 52 06 09 c1 1d 50 53 e6 62 eb 18 7f a8 48 94 c5 fd 5e 46 64 4a 9e e7 d2 7b dd cb b7 18 fd 03 be ad 51 40 e0 ed c9 bc fe 8f f2 ec 8a 54 83 3c 3b 63 d7 47 45 0e 79 e1 2f 60 92 04 96 df f8 24 98 47 50 93 92 1d 0e 2d 6d 7d a2 8f 97 03 5b ed bb 0d f3 bd c7 6e zm_remote_camera_rtsp.cpp 318
2016-03-16 21:41:37.674313 zmc_m2 31569 ERR Error while decoding frame 21445 zm_remote_camera_rtsp.cpp 317
2016-03-16 21:41:37.343278 zmc_m2 31569 ERR 256: 00 00 01 21 e0 00 22 3f ff b5 b4 83 0f 27 e7 24 92 a2 19 1d ed c2 ef 05 05 09 08 a8 a3 a0 a6 bd c0 5e 2a 83 cb 6a 2c 70 2e d3 c1 4f 4b 70 14 46 72 f1 d3 1f c9 9c d3 ec c8 b4 e0 47 d4 87 5c 06 bc b8 ab fe 0d cf fb 6b 05 9d 60 90 88 ec f8 17 dc 62 bb 98 30 14 e8 16 94 55 a2 10 b2 df 90 ba bd 87 06 d1 7c f1 26 26 73 ea 5d 6d 73 a0 53 cb 9a df 1e ff c6 0b 44 e6 b8 76 6c 62 2d b1 8b 05 37 6f 0d 8c 15 71 e9 7f 49 da 8f c7 07 48 03 5a 00 3a 63 6f 77 a2 b4 80 64 88 56 9e ac ce e7 35 fe 8b 70 db 32 38 8e 80 cd 88 1b 22 54 8e e6 aa 7e 70 71 05 f3 0c 14 9c 77 98 e4 4b 70 45 84 8c 7f 8d 9f 37 27 88 db c7 b1 b3 22 b9 fd 62 f8 be af a0 62 e3 8a c4 5f d2 ec c4 ef ef c7 de 39 55 6d 7a 44 25 1b 55 3d 4a 35 fa 56 0a 67 05 af 9e 2e cb 61 7e 79 46 d4 cb c9 de d2 f6 fc 9d e4 d7 zm_remote_camera_rtsp.cpp 318
2016-03-16 21:41:37.339499 zmc_m2 31569 ERR Error while decoding frame 21445 zm_remote_camera_rtsp.cpp 317

What do we do next?

Ok so maybe it is something to do with the inability to decode frame, but it is going to be a pain to find as it is not instant. But ffmpeg and remote both use the same ffmpeg decode. It is weird it doesn't recover after the next i frame.
I would also try libvlc monitor as I don't think that shares the same h264 decoder.
Then I would be running ffmpeg outside zoneminder to see if also has the same issues with these cameras over the long term. Ensure you are on the same ffmpeg version as on the zm machine If this does show up the issue then try updating to the latest ffmpeg. At some point we might have to debug, but I really think it is something in the camera that is either non-conformant or just obscure and ffmpeg h264 decoder doesn't support.

Sure looks like some bug to me - with exception for the first crash, zmc aborts after exactly the same amount of runtime: 2h16m32s (roughly 25400 frames).

Date/Time Delta Component Server PID Level Message File Line
2016-03-17 06:47:46.302242 02:16:32 zmc_m1 7607 FAT Unable to decode frame at frame 25464 zm_ffmpeg_camera.cpp 178
2016-03-17 06:45:18.739751 - zmc_m1 7607 INF Garagedorr: 25000 - Capturing at 3.13 fps zm_monitor.cpp 3131
...
...
2016-03-17 04:36:40.833082 - zmc_m1 7607 INF Garagedorr: 1000 - Capturing at 3.07 fps zm_monitor.cpp 3131
2016-03-17 04:31:14.378396 - zmc_m1 7607 INF Priming capture from rtsp://192.168.1.131:554/track2 zm_ffmpeg_camera.cpp 104
2016-03-17 04:31:14.372999 - zmc_m1 7607 INF Starting Capture version 1.29.0 zmc.cpp 250
2016-03-17 04:31:14.108529 02:16:32 zmc_m1 7234 FAT Unable to decode frame at frame 25284 zm_ffmpeg_camera.cpp 178
2016-03-17 02:14:42.134702 02:16:32 zmc_m1 6710 FAT Unable to decode frame at frame 25290 zm_ffmpeg_camera.cpp 178
2016-03-16 23:58:10.330093 02:16:32 zmc_m1 5953 FAT Unable to decode frame at frame 25292 zm_ffmpeg_camera.cpp 178
2016-03-16 21:41:38.203150 01:56:48 zmc_m1 31484 FAT Unable to decode frame at frame 21634 zm_ffmpeg_camera.cpp 178
2016-03-16 19:44:50.279843 - zmc_m1 INF Starting Capture version 1.29.0 (first with ffmpeg)

This morning I do see live feed on monitor 1 (ffmpeg) without having made any change since the flickering started around midnight. Monitor two (Remote) is still with steady 100% grey.

Will try libvlc now.

EDIT: No luck with libvlc... see below error message when trying. Any idea?

2016-03-17 08:07:34.368088 zmc_m1 9535 FAT Unable to create libvlc instance due to: (null) zm_libvlc_camera.cpp 170
2016-03-17 08:07:34.352626 zmc_m1 9535 INF Priming capture from rtsp://192.168.1.131:554/track2 zm_libvlc_camera.cpp 145
2016-03-17 08:07:34.346179 zmc_m1 9535 INF Starting Capture version 1.29.0 zmc.cpp 250



Do you think its any idea to tune the camera compression settings or try MJPEG instead of H.264?
I'm capturing off stream 2 (H.264 Main Profile, 640x360, 3 fps, constant bit rate 1Mbit). As you can see on the hi-res stream there are additional settings available when going for variable bit rate, e.g. the I-frame interval can be chosen between 1, 2, 3, 4 and 5 seconds).



EDIT: I tried swapping from H.264 to MJPEG on monitor 2 (and iterated Remote, ffmpeg and libvlc) but didn't get it working; this is what I got with Remote (apart from tons of "Error decoding frame 0")

2016-03-17 08:26:45.816789 zmc_m2 10489 ERR Error while decoding frame 0 zm_remote_camera_rtsp.cpp 317
2016-03-17 08:26:45.261601 zmc_m2 10491 WAR The device is using a codec that may not be supported. Do not be surprised if things don't work. zm_rtp_source.cpp 69

Weekend update
libvlc didn't work, or my setup is missing something?. I haven't tried any long term test with ffplay as I do not have any monitor connected to this Raspberry (yet). I could use a remote screen via ssh -X but have opted out on that for the time being as it would add additional factors to the game.

But as a test I changed from "fixed bitrate / H.264 Baseline" to "variable bitrate high quality / H.264 Main Profile" on my second camera (still keeping it as remote) before the weekend and it has been running since then with only occasional decoding issues in the log, monitor still running. Also the first camera (still on ffmpeg and fixed bitrate since my failure with trying libvlc on Thursday) has run without issues since over the weekend.

Variable bit rate / high quality has increased the network and CPU load somewhat, but capturing seem to have less issues with less compression in camera 2? I'm now reducing the compression on the first camera (on ffmpeg) as well to see where it takes me.

Tarde, sed tute!

Steve: do you have any comments to my latest findings?

Every camera is different so it could just be something that it does that no others do. The other thing that struck me is sometimes cameras go grey when people hit memory limits, in your case some leak or just time based growth in ram usage? So try a single camera that is reliably failing, as the only camera on the box to see of it gets better. But if they are running all weekend this could take a while to nail down.

Or maybe it is something in ffmpeg with this particular h264 stream, then trying a later ffmpeg might also be an option. ffmpeg 3 works but has lots of deprecation errors that I have a pull request in to fix.
Wish I could help more but short of buying the camera it is hard to find something so inconsistent in occurrence. You could try cranking up the debug logs some more to see if you get any more information, but that will hurt performance.

Thanks for your thoughts Steve, I really appreciate your feedback.

I'm new to ZM, but I've been working with UNIX/Linux for the last 20+ years. What memory figures should I concentrate on?
/dev/shm has been mentioned in other threads, my Raspberry Pi 2 is rather "fixed" on that, 1 GB RAM and a SD card to run OS from.
I've mounted /var/cache/zoneminder and /var/lib/mysql on an external disk as I've learned the hard way that using the SD for permanent & frequent storage of files from an RPi application is usually a bad idea...

This is from my machine; /dev/shm usage is reported to 43% (~200 out of 460 MB in use)
A snapsbot of top look like this, can't say I've noticed any growth of any process so far... the largest monitor is capturing 1280x720 from a camera which I have had no issues with so far. The grey out took place even if removing this (in CPU and memory largest monitor).

I intentionally left a montage open to have some zms activity on top of zmc and zma, the problem has not struck since last Thursday.
Also, problem has not struck during streaming, only at times when only capturing and analyzing is taking place.

Today, I saw this in the log on the ffmpeg monitor, seemed to have fixed itself without any zmc restarting?

Is it only a coincidence that zmc 2 had a glitch more or less exactly 30 seconds after zmc 1's feed were restored?
Some issue with the shm or semaphores between the monitors, etc?

2016-03-21 15:34:51.446713 zmc_m2 10679 WAR 52: f4 da 74 21 3a 6c 16 dd 95 ab 31 0d 7a 5d 5c 2a 5d 3d a7 bd 30 f7 59 10 7c a1 71 af 1c 8d 78 68 7a b2 42 32 88 ca cc 08 50 41 8e 40 4c 62 fc 76 0c d2 e8 4c zm_rtsp.cpp 799
2016-03-21 15:34:51.438661 zmc_m2 10679 WAR Unexpected format RTSP interleaved data, resyncing by 52 bytes zm_rtsp.cpp 798
2016-03-21 13:04:49.308181 zmc_m2 10679 WAR 32: ac 5d 2b 10 e6 b1 b7 20 db a9 e1 1b 23 b6 da b4 d1 0a c3 51 c3 5d 7f 8e 66 88 69 ab c4 52 04 0a zm_rtsp.cpp 805
2016-03-21 13:04:49.304385 zmc_m2 10679 WAR Unexpected format RTSP interleaved data, dumping 52 bytes zm_rtsp.cpp 804
2016-03-21 13:04:19.288207 zma_m1 9679 WAR Signal: Reacquired zm_monitor.cpp 1420
2016-03-21 13:04:18.883923 zmc_m1 9667 WAR Buffer overrun at index 33, image 1133433, slow down capture, speed up analysis or increase ring buffer size zm_monitor.cpp
...
2016-03-21 13:04:14.989612 zmc_m1 9667 WAR Buffer overrun at index 29, image 1131579, slow down capture, speed up analysis or increase ring buffer size zm_monitor.cpp 3100
...
2016-03-21 13:04:11.388801 zmc_m1 9667 WAR Buffer overrun at index 30, image 1129880, slow down capture, speed up analysis or increase ring buffer size zm_monitor.cpp 3100
2016-03-21 13:04:11.271240 zma_m1 9679 WAR Signal: Lost zm_monitor.cpp 1420
2016-03-21 13:04:11.244623 zmc_m1 9667 ERR Unable to read packet from stream 0: error -1606004923 "End of file". zm_ffmpeg_camera.cpp

Update: I've run both my ACTi cameras on variable bitrate / high quality / H.264 Main Profile for more than 10 days now without a glitch...
The log is also much more calm now; no decoding errors.

I've now swapped back monitor 1 to "Remote" again.

Here is a process comparison between two zmc:s (row #1; monitor 2 after more than 10 days of flawless running. row #2: monitor 1 just after changing back to Remote): Very small differences in memory usage...


Hopefully, we can close this problem out soon...

@SteveG: Soon after reverting monitor one to Remote from ffmpeg at about 1600 hours today I got into trouble again...
... at 2117 hours I turned on my HiFi stack (including a Sonos Connect) the (R)STP protocol of my main router had to counter a network loop (which is business as usual if you have two or more Sonos units on a mutual wired connection).

After the network stabilization (lasting less than 30 seconds) ZoneMinder continued to have problems until I reverted monitor 1 back to ffmpeg again...

Given this and my previous input (why two different monitors seem to jam with each other at some 30 seconds interval) I strongly suspect some problem in the Remote/RTSP handling of ZM.

I have attached the logs

I can understand that some may argue why I have cams and Sonos on the same network - I guess I'm just like most people at home; you buy what you can afford and you won't start off with a complete new set of network gear for ZM & your cameras...


FYI: I've turned off the WiFi of my Sonos in the HiFi stack (I only kept it on due to that another Sonos unit without copper connection is just next door; but I seldom [never] listen to music there when my HiFi next door is powered up...)

After some time I will turn back monitor one to Remote and see if the problems come back... but I'd appreciate any feedback on the problem...

Not solving your problem but can you check options->system, have you got OPT_FRAME_SERVER flagged?

Yes, I do have the frame server daemon enabled.

I chose this since I'm running on a "slow" hardware (Raspberry PI 2) with storage on an external USB disk (mechanical, 2.5"). I don't know the speed of the USB-bus of a Raspberry, but since power is low I suspect it to be slower than in a traditional hardware.

Should I turn it off?


Config update:

I could not rule out my HiFi-stack's Sonos contributing to the problem each time I turned on or off the stack (typically once per day) so after disabling the stack's Sonos WiFi I reverted back camera 1 to "Remote" some day later. All still run (~5 days so far) without the problem showing up! I still do get occasional capture errors (Unexpected format RTSP interleaved data, Unexpected channel selector 183 in RTSP interleaved data).

To be honest; I have not seen any 100% grey in a very long while now.
Perhaps it was connected to the higher camera compression I started off with?

I also want to increase the resolution, so I will probably soon change from 640x360 to 1280 or more on one of the cameras to see how much more juice it burns off my Raspberry.

Again, I wish to thank you Steve for tending to my problem.

Just to add a few points, to increase understanding of what is going on:

When you can from remote to ffmpeg, or make any other change, or just click Save on Monitor properties, the zmc and zma and zmf processes are killed and restarted. So the problem goes away for a while.

As to zmf... I have not found any benefit to using it. I may actually make an effort to remove it in future.I recommend not using it. However it should not have an effect on your problem.

The difference between remote and ffmpeg is that remote uses our own home grown rtsp client which gets the frame data and passes it on to the ffmpeg decoder. The ffmpeg monitor type just hands the ffmpeg library the rtsp url and the library handles parsing the rtsp stream. Long term I suspect that we will deprecate our remote rtsp based code as I would assume that ffmpeg's rtsp code will be better maintained than ours.

It sounds like the code is not recovering well from packet delay.. I can't imagine why.

You might consider turning on debug logging.

Good news is that when we release 1.30, your pi will likely benefit greatly from the h264 video passthrough code....

Thanks for the feedback iconnor.


I've now disabled zmf and restarted the service.

We'll see how things turn out; I'm eager to place an order for my third ACTi cam to complete the setup around my house...