Thousands of Blue Screen events on remote IP cameras

[fabietto]

I join to post ... I also have the exact same problem with a camera Apexis/Foscam!

[bb99]

I'd be real interested in hearing what effect increasing ring buffer size has on this problem:

Buffers Tab

Image Buffer Size
This option determines how many frames are held in the ring buffer at any one time. The ring buffer is the storage space where the last ‘n’ images are kept, ready to be resurrected on an alarm or just kept waiting to be analysed. It can be any value you like with a couple of provisos, (see next options). However it is stored in shared memory and making it too large especially for large images with a high colour depth can use a lot of memory. A value of no more than 50 is usually ok. If you find that your system will not let you use the value you want it is probably because your system has an arbitrary limit on the size of shared memory that may be used even though you may have plenty of free memory available. This limit is usually fairly easy to change, see the Troubleshooting section for details.

[Briago]

From what I have seen, I think it is related to the amount of time it takes the image to transfer over the network. I think if zoneminder is still receiving the previous image, it may still request the next image from the camera. When that happens I don't think zoneminder handles two images coming in well, or maybe it gives up on the first image and makes it blue? I find that if I speed up the frame rate too much I start to get random dropped frames, and if I go high enough all frames are blue.

You may have a fast network bandwidth wise, but sometimes small interuptions in the network (especially wireless) cause short delays (lag) (were talking about fractions of a second here).

For all of you with the blue image issue, please test running at various fps and see if it impacts the performance.

If this is the problem, maybe zoneminder could be adjusted to handle receiving multiple images at the same time and still log it in the database accuratly.

Of course this is all pretty much a guess, I don't know how the code actually works, or if it is possible for this to be the problem... Just an idea.

[bb99]

All a function of the ring buffer: more data=larger ring buffer (store and forward, fifo, etc). Pay close attention to the memory caution in the quote or you'll experience a whole new set of problems.

[Briago]

I don't think it is the same. The ring buffer maintains a group of images such that if an alarm comes you can record the already occured images. However, the managing of downloading of images into the ring buffer is where I think the issue is.

The ring buffer will hold x images, but I think it still depends on one image's download being complete before the next image starts to download. If you have a frame rate of 10fps, then every single image needs to complete its download in 1/10th of a second (normally not an issue). However, if a network disturbance occurs and you have a small lag, that image could be delayed, and cause it to be made into a blue screen.

If this is the issue, the settings for the ring buffer would have no effect, as it is the management of the downloads that is at issue.

[bb99]

The ring buffer is the storage space where the last ‘n’ images are kept, ready to be resurrected on an alarm or just kept waiting to be analysed.

Anyway I'd be interested to hear if increasing it helps any.

[DarkBeer]

When I look at my camera setup, the Max FPS field is blank. I left it that way in 1.25 because that was the same way I had it set in 1.24. Again, same number of cameras, identical zone setups, same network, but on the faster computer running 1.25, I get these errors, whereas the slower single core CPU computer, with a system load of over 6 doesn't get the blue screen issues. Doesn't make sense.

[bb99]

It might make sense when looking at the change log. In 1.24.3 mmap became the default memory manager as a "feature" instead of shared memory. That's why I'm asking the question. Oh by the way: I can get dropped frames by reducing the ring buffer on a shared memory system.

[DarkBeer]

What setting are you using? I copied the same setup to my 1.25 install, so on the buffers tab, image buffer size (frames) is set to 40 on each (5) camera. The quote you had above seems to suggest 50 would be an upper limit. I can't help but think the faster system I am now running on shouldn't need a larger setting than the older system. Is it possible it is too high and I should try lowering it? Appreciate the continued feedback and advice. I'd really like to get this working on the new computer so the other can be retired, but I have to get rid of all these false alarm recordings.

[Briago]

DarkBeer, if the FPS is blank it will use the max possible. That should mean that it would wait for the images to come in I think, but I suggest to try setting it to 3fps, 7 fps, 12 fps, and 20 fps. Let it run for like 5 minutes each, and see if there is a difference. Also, if it creates blue images even at the lowest fps, then experiment with the other cameras disabled.

It may not make sense thinking about it right now, but I'd try it and at a minimum it will give you more information to work with.


Did you try increasing/decreasing the ring buffer also?

[DarkBeer]

Finally have some time to start tinkering with this again. I have image buffer set to 50, which I read as a suggested max. Warmup frames is 25, pre event image count is 10 and post event image count is also 10. Max FPS is set to 7. All 5 Foscam FI8910W cameras are set the same way. Still getting blue frames on each camera at random intervals. CPU load on this computer is .51. Same 5 cameras, running on older single core CPU produce a load often over 6.0, but no blue frames. Max FPS is blank, so in theory they are recording even more frames. Buffers setting on that computer is 40, warmup frames 25, pre event 10, post event 50. Old system is running 1.24.2. Don't see how it can be a wifi issue, since I'm not changing anything between the 2 systems. Faster computer should be able to process the buffers even quicker than the older computer. When I first setup buffers on the new computer, I duplicated the setup of the old. Does it make sense to take the buffer even lower than 40? The old computer is difficult to use for anything else since the CPU load is so high running ZM, so that is why I am trying to move to the new computer.

[Briago]

Sounds like it is a different issue. Just similar symptoms.

All I could suggest at this point is to be very systematic in your trials. Pick a netural setting for every setting and change only one at a time (extreme high then extremem low). See if any settings make any difference. Beyond that, no ideas. I don't know enough about the code to suggest any potential bugs in the code...

I do recommend to only have one camera live while doing the tests (in case the setting of another camera also causes the effect on the camera you are testing).

I assume you have reconfigured the logging to be more verbose, and have looked for any error messages indicating the lost frames?

[DarkBeer]

Unfortunately this Foscam camera produces a constant corrupt JPEG data error in the logs. They fixed this on another model with a firmware update, but haven't fixed this model yet. I'll just keep tinkering and hopefully figure it out at some point. Appreciate the suggestions!

[Briago]

Sounds like a plan. If you find something out, let us know.

[Flasheart]

In a shell session from the same server, start pinging one of the problematic cameras and leave it for some time, perhaps a day.

Come back, hit ctrl+c and look to see how many dropped packets. Anything other than 0 means there's something wrong with the network, camera or server's connections and that can be diagnosed.

If the network is perfect, I would then try alternative feeds, depending on what the camera can support. static jpg is the most robust and also slowest, then mjpeg, mpeg4, h264 - to see if one type of feed is more reliable.