Signal loss causes blue screen that requires camera reset

[empty01]

The frame rate is 4fps on all cameras, ring buffer is 32, 12 warmup frames, 16 Pre Event Image Count. Currently 11 cameras, but /dev/shm is currently at 40%, so memory load is unlikely to be the problem. besides, I tried switching off all but the 4 most troublesome cameras and had /dev/shm down to 15%, and still got the blue screen problem.

I also tried switching down the colour space; not a particularly extensive experiment but I still got signal loss / blue screen issues with 24bit.

The key point here is that I can view the camera's RTSP stream on VLC, and view its stream via its web interface, even though Zoneminder is giving a blue screen. Zoneminder is losing the stream - which is not that shocking (could be signal interruption / power drop etc). But why isn't Zneminder picking the stream back up? That seems to be the key issue.
Cheers
Mark

[knight-of-ni]

Remove the 5 you have in the analysis fps field. It won't help you if the camera's fps is less than that value, and I would not be surprised if it behaved badly in that condition. It is not as well tested, and unfortunately the author of that feature is no longer living. I suspect it could use some love.

Note that I asked the value of your system load, not consumption of /dev/shm. The value is displayed at the top of the web console. Since that value can fluctuate over time, I'm not looking for a real precise answer here. The important thing is you need to allow sufficient headroom for other tasks. For a dual core cpu, you don't want your load to go beyond ~1.5 for an extended period. If the system load is too high, one can get the same symptoms you describe. That is the key point I am driving towards.

You've got 11 cameras. Are they all 1080p? That's a bit much for just a dual core processor, even at 4 fps.
You can easily test if the system load is too high. Disable all your cameras (function=none, uncheck enable), except for 3. Then just wait and see what happens. If your system remains stable, then add two more cameras, and wait again. Repeat until the symptoms return.

I don't understand your response concerning the colorspace. Your screenshot shows the colorspace is set to 24 bit, and I pointed out you can get a performance boost by increasing that to 32 bits. However your response seems to imply I said something other than that.

[empty01]

Hi knnniggett

OK, apologies for the misunderstandings.

RE having 5 in the analysis fps field - I actually added that as an attempted fix after reading some other post - the problem was there before I added it. However, if it isn't going to help and may make things worse, i'll remove it - thanks for the heads up.

RE the colour space - OK, i'll bump up to 32bit (all cameras were at 24bit). But wouldn't that increase the memory requirements (just a guess)?

RE 'system load, not consumption of /dev/shm', I see. Well my load is usually well above ~1.5. At the moment it is at 7.38 but it usually hovers between 3 and 8. Humm? Could that be it? How could I bring it down (other than by having fewer cameras)? The box is only used for Zoneminder. I did set aside 70% of the memory as mapped for Zoneminder, following the instructions under '' in the Documentation FAQ.

I'm trying another possible fix at the moment: I put my switch into managed mode and then switched off the 'Green Ethernet' options, which are:
•Energy-Detect — Auto-detection of inactivity on a port, and subsequent turning down of transmit power. It may take about 1 second for the cable to power-up when it becomes active again.
•Short-Reach — Reduction of power over Ethernet cables shorter than 40m. The cable length is detected automatically with an accuracy of +/- 10m.

Its been about 12 hours since I did that and none of the cameras have blue screened yes. I'm not optimistic and will certainly try your suggestions once one of them does blue screen (and write another whining post

All best wishes

Mark

[knight-of-ni]

Yeah, the analysis fps field can't help you in this case. It can only hurt, so we want to leave it empty.

Yes, bumping to 32bits is going to use more memory, but you have 6GB of ram and your bottleneck is cpu resources. This is a fair trade off. You've already increased the default size of your /dev/shm. Keep an eye on it as you bump the colorspace up, but I don't expect you to run out.

Generally speaking, the system load on any unix machine is a 1:1 representation of how busy your cpus are. In the simplest case, a single core cpu, with a load of exactly 1 is 100% full. Any additional load on this system will just back up. Processes will not be complemented on time, and the load counter will increase to reflect that. As the load increases further, processes will begin to time out. From this, I'm sure you surmise a dual core machine hits 100% at 2.

Additionally, you have to leave room for normal operating system tasks so realistically, the average load has to be less than the number of cores you have, on any machine running realtime tasks. A load of 7.38 means the system is hopelessly behind. If you leave the load that high, you can't expect your system to remain stable. I'm a bit surprised you haven't experienced any issues more severe than cameras simply timing out.

Different ways to solve the load problem are:
- get a faster server. Generally speaking, a machine with more, slower cores is better than a machine with fewer, faster cores
- get an additional server to split the load
- lower the resolution on your cameras down to 720p. -> This will dramatically reduce the load.

I don't know if you are in a position to consider different hardware, but a core 2 duo is a fairly old. I'm sure you know that. Nearly any machine you can buy today, even a cheap sff desktop, is going to perform better.

For example, I'm using an HP Slimline 450 at my mother-in-law's, and I've posted in a table at the bottom of this blog what you can reasonably expect in terms of cameras: http://zoneminder.blogspot.com/p/hp-slimline-450.html

And just for giggles, if you google the cpu in the slimline versus a core 2 duo "intel E8400 versus J1900" you can see that a modern four core celeron cpu will perform better: http://cpuboss.com/cpus/Intel-Core2-Duo ... eron-J1900

Another example, and the absolute best in terms of bang for the buck, is to pick up an off-lease rack server off ebay. You can get quite a beast for just a few hundred dollars. The downside is size, noise, and appearance.

[empty01]

Oh well - back to black (well, blue). 3 monitors went to blue screen in quick succession. I guess it wasn't the switch 'Green Ethernet' options.
I've now switched to 32 bit colour space, and my /dev/shm went up to 54%. I guess my puny processor just isn't up to the task. Was hoping to avoid having to get better computer hardware.

I appreciate your points about the load, but while my load is high the system still seems to be performing fine until 1 or more cameras blue screens - so why can't Zoneminder pick the streams up again? I mean, if I reboot the cameras everything seems to work fine again. Is there not some cron job I can set up to restart some component of Zoneminder / SQL / Apache so it picks up the streams again?

I don't mind the idea of a rack server and its all placed in a seldom-accessed place. BTW, what does 'off lease' mean? I work at a large university and could ask if they have any they are no using any more.
Thanks again
Mark

[empty01]

Here's an even more bizarre observation.
When I restart the computer Zoneminder is running on, exactly the same cameras that were giving blue screens before I restarted give blue screens after I have restarted. That's restarting the computer, not just restarting Zoneminder!!

The same observation when I restart the switch.

This must mean that Zoneminder or the switch has affected the cameras.
And yet I can view the bluescreened camera's streams on their web portals.
However, for the blue screened ones I can't now view their RTSP streams via VLC (I can do for the non-blue screened ones).
So it seems to me that something is knocking out the camera's RTSP streams.

Any thoughts?

[knight-of-ni]

It makes perfect sense.
Your machine can't reconnect to the camera because it is too busy trying to manage the load of the other cameras... then like I said previously... the process times out. In this case 10 seconds goes by and then the system thinks the attempt has failed, so it tries to start over ....and over .... and over....

When you restart, you are starting from a clean slate. It's hasn't yet gotten stuck in that loop. At that moment, motion detection has quite started and events are yet being written to disk. By design these tasks cause spikes in your cpu load when the system detects events.

You're not the first person to show up here, who has loaded his system up far beyond what it is capable of doing, then has wondered why it isn't working properly. The system load is your measuring stick on any system running any application. Go beyond that and you will have troubles with any application trying to do something in realtime.... Loading 11 instances of vlc might yield the same kind of problems, but then again vlc isn't trying to do motion detection or write events to disk.

Large corporations don't wait for their servers to fail before replacing them. Instead they lease their servers. When the lease expires, they give the servers back from whence they came, even though there is nothing wrong with them. This creates a surplus of off-lease servers, which resellers put up for sale on E-Bay. The cost savings is tremendous.... $300 will get you a beast of a server with many cores and many GB of ram which was originally purchased by some business for several thousands of dollars.

[knight-of-ni]

Here is one example:
http://www.ebay.com/itm/Supermicro-2U-X ... 361b882302

You get 8 total cores and 32GB ram for just ~$150. You will need to add your own drives, but if you use software raid, or ZFS, you can get away with any hard drives you've got lying around. Mix and match.

There are many more of these kinds of servers for sale on ebay, many of which are even faster, more cores, more ram, more drive slots, etc.

While you could certainly get away with a modern desktop pc in your case, the cost to do that would be right around the same price as the server, if not more. And you would end up with less horsepower.

[empty01]

Hi knnniggett
Thanks for explaining off-lease - i'm in the UK - 2 nations divided by a common language
I like the idea of getting a chunky server as I do have more cameras to add, and as I said, I work in a large University so I suspect I could find one going free / cheap. I'll definitely chase that up.

But, if you'll forgive me for pursuing this:

• I tried disabling all my cameras in the 'Function' column except the one that is currently giving a blue screen; got the Load down to 0.2, and /dev/shm down to 5%. I also stopped and restarted zm - and still I get a blue screen.

• Also, I can't view that camera's rtsp stream via VLC.

As soon as I reboot the camera from its web portal, I can:

• see it in zm

• View its rtsp stream via VLC.

Unless I am missing something fundamental, that tells me its a camera issue - the camera's rtsp stream has gone down (although I can still view video via its web portal - which I guess must be using something other than rtsp).

So I guess this isn't a zm issue after all (although I take your point that my Load is too high and I will get other problems unless I solve that).

I've only use RTSP in Zoneminder. Since the camera's web portal can see the video, presumably there is another way (other than RTSP) to view it in zm. Do you have any suggestions.

Thanks again for all your help, and apologies for the additional questions.
All best

Mark

[empty01]

Hi knnniggett
I took your advice and got myself (for free) a Sun Fire x2250 server (2 x 4 core Xeon 5400; 16Gb RAM) and sure enough no more blue screens (after 3 days), so thanks for the advice.
Just a recap - with 8 cores should I now be trying to keep my load below ~6? At the moment it hovers around 2.5 to 3 but I want to add more cameras.
One thing I didn't realise is just how much power these servers consume. Any thoughts on how I can keep that down to a minimum?
All best
Mark

[Baylink]

Big trucks consume lots of fuel, even at idle. I am presently using a Dell sc1430 - that's a 2 gigahertz dual quad - to run 13 cameras, and my load average runs around 13, Which is higher than I think they recommend, but it seems to deal with it magnificently. I don't get any blue screens or streaking except on one camera where I think the run is maximum distance.

[empty01]

They sure do. Haven't set up all my zones yet so i suspect my load is going to go up a lot as I will have about 15 cameras.
Still, no blue screens, so am happy

[Baylink]

It's probably worth pointing out to that my system has one big Zone on each camera. That cuts both ways, of course.

[empty01]

Arghhh! I spoke too soon. After taking knnniggett's advice and getting a server with lots of cores, I am still getting the blue screen problem!

As my post on Sat Sep 09, 2017 9:49 pm explains, while having a big chunky server is appealing, its doesn't logically explain the problems I am experiencing. Somehow Zoneminder is knocking out the camera's RTSP stream.

I'm on my knees here. I've invested a lot of time and money into a Zonemider-based system, but a problem I have experienced with 3 different computers and 4 separate system installs, on a range of cameras, just won't go away.

All best

Mark

[Baylink]

Have we addressed which version of ffmpeg you're using, and where it came from?