New problem - camera goes red

[Basildane]

In the past 2 days a new problem has appeared in my installation.

Camera 1 just goes RED, for no apparent reason, and never recovers.
I have to manually reset it in ZoneMinder.

The camera itself is fine, and I can connect to it outside of ZoneMinder and it is streaming normally.
What's really odd is the system has been up for over a month, and then this just suddenly started happening.
UGG!

This is the error from the log:

Code: Select all

Unable to read packet from stream 0: error -541478725

/ZoneMinder/src/zm_ffmpeg_camera.cpp  Line 215

[Basildane]

No one has a clue?
I get up every morning to a random group of cameras just RED. No recordings of alarms overnight. I have to manually go into zoneminder and edit each camera and click SAVE to get them to reconnect. I have to repeat this ritual every single day.

My old Atrix DVR ran for 4 years non-stop and never lost connection to a camera. Never.

The cameras are fine. I can see them with other software, including when they are RED in zoneminder.

This is killing me. Not the pain of having to restart all the time, but MISSING recordings. If something happened, I might only have a RED jpg to show the cops.

There is some network glitch or something, and then zoneminder does not recover. This is a very serious problem if anyone wants to use this software for important monitoring.

[mikb]

Although it's obviously causing you a lot of grief, the fact that the forum isn't overrun by people reporting this, or pitching in to say "known problem - do this" means it's something fairly rare, possibly unique to your setup.

I know that's no great help to fixing it, but "This is a very serious problem if anyone wants to use this software for important monitoring" isn't fair or accurate

Zoneminder is a persistent little bu**er when it comes to IP cameras dropping off the stream, and re-getting them. There may be a short pause before it retries, but it comes back.

I don't know whether your use of FFMPEG might be complicating things. Not pointing any fingers, just saying, it's an extra layer of stuff to go wrong. What version of FFMPEG are you using?

You haven't said what camera brand you are using -- might be relevant, and also, if this suddenly started happening: What did you change Something totally unrelated may have upset things, upgraded a library?

And have the cameras, and the PC, been given a power-down-reboot since it started (as mine worked fine until they'd been up for a couple of months, and then started, EVERY ONE OF THEM, dropping connections, faster and faster, until almost unusable).

[sogood007]

I have similar problem for Trendnet TV-IP672 camera but don't have it for my other Swann (HIkvision clone) IP camera. It happened randomly and when the problem pop up, I need to restart the camera or eveb restart zoneminder I haven't spent much them to isolate and debug the issue. I am using ffmpeg source

[Basildane]

mikb, the ffmpeg version appears to be git-2013-10-09-885739f. I compiled from source. Only the ffmpeg channels have this problem.

All networks will have intermittent glitches, so we need to find a way to make the zm agent gracefully reconnect if something goes wrong. And that's just an assumption, the other possibility is a bug that is causing the agent to go wrong.

I have ruled out the cameras by monitoring them with another application, and they are online and streaming.

I can get everything working again by simply going into settings in zm and then clicking SAVE without changing anything. This proves without a doubt that problem is in zm. It simply does not repair itself after a failure. I'm willing do everything I can on this side to troubleshoot and help fix. But this is a new application to me and will take time to learn the code.

Although ffmpeg definitely is not perfect, it cannot be held responsible for this problem because of the facts above. Logic leads to only one conclusion. (I do believe ffmpeg is causing the smearing problem, but that's an entirely different matter).

The other answer to your questions: the cameras are Grandstream gxv-3672HD's.
The problem has occurred twice already today alone.

[martinanth]

Just a thought, could you clear any entries you have in the Maximum FPS and Alarm Maximum FPS fields (make them blank).

My camera's were going red in Zoneminder (but streaming via VLC worked fine), by clearing these fields, ZM has not missed a beat since.
The cameras are already set to 15 FPS internally (not via ZM).

Virtual Centos 6.4 server
3 * Dahua IP-HFW2100

[Basildane]

Just a thought, could you clear any entries you have in the Maximum FPS and Alarm Maximum FPS fields (make them blank).

My camera's were going red in Zoneminder (but streaming via VLC worked fine), by clearing these fields, ZM has not missed a beat since.
The cameras are already set to 15 FPS internally (not via ZM).

Virtual Centos 6.4 server
3 * Dahua IP-HFW2100

Thanks for the tip, I'm going to do that right now.

Also, mikb, forgot to answer one more of your q's. There were some changes lately, being updates to the linux installation. Every week or so, I try to apply updates to the servers...

[Basildane]

Oops, I responded to quickly.

max fps fields are already blank here. I found that out back when I first started setting it up.
I'm going to start looking at the source code tonight...

[Basildane]

I was just able to get a minute to test something. I deliberately interrupted a camera for a second.
ZoneMinder lost the image, and went red, and never resumed streaming.

Simultaneously, I was watching the camera with GSurf Pro. The camera froze, then resumed streaming in a couple of seconds.

Now, how to get the zm agent to resume streaming after an interruption...

>>> correction

Further testing revealed that zm did recover several times. Even up to 10 seconds of data loss. So I recall my last test as inconclusive...

I'll keep testing.

[EdwinLandy]

Just wanted to give my experience for any troubleshooting. I saw a Red screen after adjusting source settings. I'm using ffmpeg, H.264, with a Grandstream GXV3615. After I went to the source page and hit "save" again, it went away. No trouble since.

Red Screen

11-19-2013 4-03-04 AM.png (79.58 KiB) Viewed 9680 times

[mikb]

All networks will have intermittent glitches, so we need to find a way to make the zm agent gracefully reconnect if something goes wrong. And that's just an assumption, the other possibility is a bug that is causing the agent to go wrong.

I can get everything working again by simply going into settings in zm and then clicking SAVE without changing anything. This proves without a doubt that
problem is in zm.

There were some changes lately, being updates to the linux installation. Every week or so, I try to apply updates to the servers...

1) That's some good extra information that may help narrow it down -- thanks!

2) You're right -- networks can be unreliable and the TCP/IP layer should hide all that, but occasionally the brokenness breaks through. The ZM Capture agent *does* retry, at least for me, when my cameras kept dropping the stream, it kept bringing them back up. Hundreds of times an hour. Persistent! If I disconnect a camera, and reconnect it, ZM picks it up in fairly short order without a reset of anything.

Why that persistence isn't working for you, I don't know. That's why I'm thinking that FFMPEG may be the complicating factor, by routing the IP camera through it.

3) Clicking save in ZM makes it work again: Does not automatically mean the problem is ZM -- I assume (dangerous) that ZM tears down the FFMPEG it was using and restarts it along with the capture demon etc. So this may not be reliable

4) If you update your system every week, that's very diligent of you, but you are standing on a moving platform, expect some wobbles along the way.

[Basildane]

Unfortunately, I cannot even start zm now.
Completely dead in the water.

Can uninstall, reinstall, nothing.

I'm not sure I want to do complete rebuild after investing 3 months getting this setup.

[ralph179]

any resolution to the red screen? I am having a similar problem with one camera. When I look at the frames of any event the first 25 are normal status and the image is visible from frame 26 on the status is alarm and the image is red. I had this camera on another system and it was working fine. Any ideas?

Zone minder 1.26.4

Ralph

[Basildane]

Over the last few days, I have managed to recompile everything on my server, from source. I have the latest ffmpeg, and everything was updated from git to the latest and greatest.

I have still had one RED event yesterday. The system "feels" more stable, but that could be my imagination.

I looked at module I suspect, zm_ffmpeg_camera.cpp. I think it would be trivial to detect the error condition, then reconnect. It will take a lot longer for me to do it, because I don't know the software structure. But I'm going to make an effort. I don't know when I'll have the time though. I'm getting pulled 10 different directions.

I think this project is cool and shows great potential. I am committed to it.

[ralph179]

So this was a new camera to my system and after adding it I was getting the red image configured with ffmepg. Iwas reading the forum and seen configurations using rtp/rtsp as the source type so I tried that but was unsuccessful. I re-configured using ffmepg and it is now working,nothing different in the setup. A little concerned it will fail again but has been OK for a day.

Ralph