Signal loss causes blue screen that requires camera reset

[empty01]

Hi
I have seen this problem mentioned on a few postings, but I have not seen any solution to the problem so far.

I have Zoneminder running on an Unbuntu 16.04 box with a 3GHz Core 2 Duo and 6 GB RAM. I have several IP cameras connected (all via CAT5E cable /RJ45) of various makes, all using Ffmpeg and RTSP streaming. All cameras are operating at low frame rates so my /dev/shm rarely exceeds 60%

In Zoneminder all cameras intermittently return a blue screen and 'signal loss' alarms (which I automatically delete with a filter). Some cameras do this less often (i.e. around once a week) and some cameras do it frequently (i.e. about twice a day). Once the screen goes blue there seems to be nothing I can do from within Zoneminder to get the image back (i've tried stopping and starting Zoneminder, disabling and reenabling the cameras within Zoneminder, and plenty more besides). The only thing that works is rebooting the relevant cameras via their web interfaces.

The odd thing is, even when Zoneminder shows a blue screen for a camera, I can still view that camera's stream via its web interface, or by using VLC and its RTSP stream. So the cameras must still be streaming and the issue must be within Zoneminder.

This is big problem as I can't reboot my cameras remotely, and its obviously a pain to have to keep on rebooting them anyway. Other than this problem I am full of praise for Zoneminder - i'm just holding off my donation until I can sort this problem out.

I have copied some error logs below - in this case 5 cameras are giving blue screens (zmc_m4, zmc_m5, zmc_m7, zmc_m15 and zmc_m18). To me this suggests that Zoneminder is trying but failing to recapture the RTSP stream.

Any help gratefully received

Best wsihes

Mark

2017-09-05 16:12:39.127628 zmc_m4 5122 ERR Unable to open input rtsp://192.168.1.187:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:39.083149 zmc_m4 5122 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:37.060078 zmc_m5 5121 ERR Unable to open input rtsp://192.168.1.189:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:37.021840 zmc_m5 5121 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:34.098500 zmc_m7 5159 ERR Unable to open input rtsp://192.168.1.191:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:34.063500 zmc_m15 4527 ERR Unable to open input rtsp://192.168.1.192:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:34.024694 zmc_m7 5159 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:34.024241 zmc_m15 4527 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:32.071890 zmc_m18 5158 ERR Unable to open input rtsp://192.168.1.136:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:32.016021 zmc_m18 5158 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:28.137749 zmc_m4 5122 ERR Unable to open input rtsp://192.168.1.187:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:28.099773 zmc_m4 5122 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:26.083394 zmc_m5 5121 ERR Unable to open input rtsp://192.168.1.189:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:26.027567 zmc_m5 5121 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:23.143379 zmc_m15 4527 ERR Unable to open input rtsp://192.168.1.192:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:23.101470 zmc_m7 5159 ERR Unable to open input rtsp://192.168.1.191:554/ch01.264 due to: Connection timed out zm_ffmpeg_camera.cpp 269
2017-09-05 16:12:23.078657 zmc_m15 4527 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441
2017-09-05 16:12:23.067474 zmc_m7 5159 ERR Open video took more than 10 seconds. zm_ffmpeg_camera.cpp 441

[empty01]

I should add, I also tried doing a fresh installation of Unbuntu and Zoneminder on a different box - entering camera details afreash, but got the same problem.

[rockedge]

same router for both setups?
I had HOOTOO IP cams that would have Signal dropout but never really tracked down the cause. They were also wired. These cameras failed completely in a short time and so I still don't know why they would have signal loss even though I could stream the cameras when using the web interface.

[empty01]

Yes, same router. All of the cameras are connected to an ethernet switch box, which is then connected to my router, and to the Unbuntu box with zm installed.
My cameras haven't failed though - as far as I can tell they are working well (i.e. still streaming).

[Baylink]

Are they PoE? How long are the runs? Does your switch have enough power for all of it?

Is your switch smart enough to let you remotely bounce a port? (Yes, that's a workaround, but those give you maneuvering room to get things sorted...)

[empty01]

None of the cameras are using PoE into the cameras themselves, but on some of them I am using PoE injectors / extractors (i.e. the extractor at the camera end). However, this had no relation to the signal loss / blue screen issue as some of the cameras are powered directly from a power supply plugged into a mains socket next to the camera, and they have the same issue just as often. I don't think the switch (a Dell Powerconnect 2824) provides PoE. If it does I'm not using it - all the power comes from mains power supplies.

The CAT5E / CAT6 runs are between 5 and 30 meters. Again, no correlation with the frequency of signal loss / blue screen issue.

I don't know if my switch is "smart enough to let you remotely bounce a port". To be honest, I don't know what that means. Its a Dell Powerconnect 2824. Its not running in 'managed' mode. Should it be?

Thanks for commenting - any further help would be greatly appreciated - this problem has been rattling on for almost a year now.

[mikb]

I think the "remotely bounce a port" comment only really applies if you are powering the cameras directly from a proper Power-over-Ethernet capable hub/switch.

Which you've indicated you're not ...

Some PoE capable switches can be instructed to turn the power off and back on, on a per port basis, which can effectively remotely reset a misbehaving device.

If you are using the basic "cheap" injector/splitters that are nothing more than 3 connectors and some bits of wire, at each end, to remap the DC feed into (and out of) the Cat-5 cable, then cable length may bite you -- it's all very well injecting precisely 5v/12v at one end, but after losses you may be scraping the voltage levels at the far end. Depends on the current drawn, wiring quality, etc.

This is why "real" PoE uses 48v and steps it down locally. Plenty of scope for voltage sag and still enough to wring 5/12v out of it

[empty01]

Hi
That all makes sense as 12volt over 30 meters could lose quite a bit. However:
(1) I have checked the voltage at the camera ends and it is still ~ 12v (I must have got lucky and bought low resistance cable, or high resistance cameras )
(2) As I said above, this can't be the issue as some of my cameras don't use PoE injectors - they have power supplies right next to where they are mounted - and those cameras suffer the signal loss / blue screen issue just as much as those using an injector/extractor.

You do raise an interesting point though. When I get the signal loss / blue screen issue, the only guaranteed way of sorting it is to reboot the camera. I have tried unplugging the power on the camera(s) for 20 seconds then plugging it back in, and while that does work sometimes, it doesn't always work (I know, I can't explain it either).

Yesterday i tried reducing the ring buffer size on all my cameras - got my /dev/shm down to <30%. Alas, this morning I still had 2 cameras that had blue screens and needed rebooting

This is all very weird really. This does seem to be a common problem as I have read quite a few posts on this and other forums mentioning it. None have provided a solution that works for me, and reading between the lines, the same is true for quite a few others that have posted about the problem. I guess it can't be that common a problem as otherwise the Zoneminder coders would have got to the bottom of it long ago. But:
(1) The problem has persisted for me
(2) The problem hasn't gone away despite installing on a different computer.
(3) while some of my cameras do suffer the problem more than others, it doesn't seem to bear any relation to the manufacturer, the type of streaming, the camera's resolution, or the means of powering the camera (PoE injector/extractor, or direct nearby power socket).

The only common denominator left is the switch. Does anybody think switching to 'managed' would make any difference?

Thanks
Mark

[Baylink]

Not really.

This sounds very much like it's just the cameras getting really confused; not an accident that screen is blue.

When the camera is bluescreened, can you still get to the control panel interface?

Is there a reboot button on that? does it help?

If so, as another possible workaround, you might be able to trigger that if a camera locks up, I think.

Do you have really crappy power?

You might want to try getting a 12V SLA battery and putting the proper barrel plug cord on it, and running a camera off of that for a while, to see if it's more stable. Switching power supplies, like those in cheap wallwarts, don't isolate you from the powerline at all.

[empty01]

Could be that the cameras getting really confused, but its happened with quote a range of makes and models. Also, I can view the RTSP stream on VLC when they are blue screened in Zoneminder. Yes, I can still access the cameras web portal when they are blue screened in Zoneminder, and I can view the stream. This suggests to me that it is not the cameras, but Zoneminder.

Yes, the only way I can reliably get rid of the blue screen problem is to access the cameras web portal and reset. That might give me another 12 to 48 hours, then the blue screen comes back.

I thought about setting up a cron job to reset the cameras every few hours or so, but I have no idea what commands to send to the camera to do that.

No, power isn't crappy - I have used good solid high amp regulated power supplies placed near the camera - same problem keeps on appearing. I even tried a 12V SLA battery I had hanging around on one of the cameras that gave a blue screen particularly often - no banana.

As I said above, this does seem to be a common problem as I have read quite a few posts on this and other forums mentioning it. None have provided a solution that works for me, and reading between the lines, the same is true for quite a few others that have posted about the problem.

[Baylink]

I don't see that you've posted a monitor config yet; are you using ffmpeg, rather than remote or other older methods?

Post a screen shot of your monitor setup?

[mikb]

(1) I have checked the voltage at the camera ends and it is still ~ 12v (I must have got lucky and bought low resistance cable

Laden or unladen?

Just checking, as it can seem really sensible to check the voltage on the DC connector, and then plug it in.

Only to find that you now can't measure the voltage any more (no access to get at the RJ45/splitter/dc jack), and it's dropped ON LOAD, and you never knew.

Off load, there's no reason for a voltage drop to manifest.

Note: Above is moot as you seem to have tried local power and battery power too

Haunted switch?

[empty01]

Hi Baylink
Yes, Ffmpeg and RTSP streaming, as stated in my original message. Here are a couple of screenshots:

s1.jpeg (74.42 KiB) Viewed 9824 times

s2.jpeg (62.18 KiB) Viewed 9824 times

[empty01]

(1) I have checked the voltage at the camera ends and it is still ~ 12v (I must have got lucky and bought low resistance cable

Laden or unladen?

Just checking, as it can seem really sensible to check the voltage on the DC connector, and then plug it in.

Only to find that you now can't measure the voltage any more (no access to get at the RJ45/splitter/dc jack), and it's dropped ON LOAD, and you never knew.

Off load, there's no reason for a voltage drop to manifest.

Note: Above is moot as you seem to have tried local power and battery power too

Haunted switch?

I checked laden by undoing the back of the camera - 11 to 12v (I'm pretty OK with electrics / Ohms law etc).
What's a Haunted switch?
As I said above, apart from Zoneminder, the only common denominator is the switch. Should I try another? Is it a realistic possible cause of the problem?
Cheers

[knight-of-ni]

What is the frame rate of each incoming stream?
How many cameras are attached to your system?
What does the average system load look like?

Note that setting 32 bit color space provides a performance boost.