CPU Load?

[bmather9]

So I have 4 x 1080p IP cameras connected to zoneminder as monitors with default 25fps. My server runs a Pentium G620 @ 2.60GHz (dual core) with 8GB of ram.

With just the 4 cameras connected as monitors (not even viewing the monitors) my load seems to be somewhere around ~2 to 3.

With all 4 cameras being viewed simultaneously in montage load climbs to roughly 8.

It seems that even if I disable the monitors, the load remains around ~2 to 3. Removing the monitors lowers the load down to normal around 0.2 to 0.5. It also seems the displayed load lags a bit; when I stop viewing the montage, the load doesn't immediately drop its hovers around 8 longer than I'd expect.

I have no recording or motion detection enabled on these monitors. I thought since I have dual core that my max load should be 2.0? Even with the load of 8, the video seems relatively smooth and fluid.

So I'm considering all options here, including a dedicated server to run zoneminder, and I really only plan to have about 6 cameras. But how would someone with say, 30 cameras, make a system like this work? Just a really powerful server? I know my server isn't that powerful, but it just doesn't seem like a setup like this would scale very well? Does everyone just settle for lower frame-rate/resolution?

It seems like it would make more sense to offload most of this work from the server, to the cameras, since they can do motion detection, recording to a network drive, etc ... all by themselves. It does seem to cause some problems for me, since I'd have to stick with a single camera manufacturer, and may lost some capability to control the cameras with automation in my server. What I'd really like is to have some cameras stop recording while I'm home (phone connected to wifi).

Any tips appreciated! Thanks!

[Linwood]

I am running seven 1080p cameras (actually running at 2048x1536). The load on zoneminder is going to vary depending on your frame rate. If you try to run them all at 25fps, it will require quite a lot of horsepower in several regards.

There are a lot of things you can do to reduce it, the simplest is to just reduce the frame rate on the cameras. I run mine now at 4fps, I've heard of a lot of others a 6 or 8. If you are using these for typical security camera use, high frame rate rarely gains you a lot.

You can also reduce the work zoneminder does by setting the "frame skip" option on the misc screen. This will let you run the cameras at high rates, but only save to disk and (I think) for the motion detection skip of similar name process only every xth frame (put in x-1 to process every xth frame). This requires zoneminder to decode every frame, and will let live viewing be faster, while saving storage and some processing load.

Obviously resolution reduction helps as well as frame rate, though I personally like the high resolution and will trade frame rate against it.

The large images also require a lot of shared memory, and so real ram, to process in addition to CPU. I found a 6 meg system just barely almost cannot handle my seven cameras (I had to increase the shared memory from 50% to about 70%).

Many IP cameras also permit multiple streams, so you can potentially run a 25fps stream to view (directly, not through zoneminder) and a slower stream to process.

The Montage view is a heavy processing view as well, as are some of the other streaming programs. They are push streams, so they run as fast as they can push, so if you have a local fast client and high frame rates, this can really consume the server (as well as the client). Lower frame rates help here also.

Your best bet to size hardware, honestly, is experimentation. My suggestion is play around with frame rate (especially) and other parameters to see what is acceptable to you, then see how it runs when you load it up. I did my experimentation in a HyperV VM first before moving to physical hardware as it was a lot faster to change around.

Off loading the motion detection to the cameras is an interesting option and depends heavily on your cameras. There is a TCP interface to zonemidner but it is highly unlikely the camera can talk directly to zone minder. More likely you will need to script some intermediate tool to tell zoneminder when the camera detects motion. If you script it yourself you are not limited to one manufacturer. I personally did not find that motion detection seems to make a huge amount of difference, though it is clearly a heavy processing load. I have been told NOT to use the separate frame server as it is reported to add more load than it takes away (somewhat contrary to the online help).

[Linwood]

Postscript: bear in mind that because (today) zoneminder saves jpg's and not actual video, if you plan to record all the time, or a lot of the time, this can make for huge storage requirements especially at high frame rates. I found I can store just shy of 2 weeks, 2pfs, seven 2048x1536 cameras on 6GB of disks. That's only 2fps. Think about that at 20fps.

[bmather9]

Postscript: bear in mind that because (today) zoneminder saves jpg's and not actual video, if you plan to record all the time, or a lot of the time, this can make for huge storage requirements especially at high frame rates. I found I can store just shy of 2 weeks, 2pfs, seven 2048x1536 cameras on 6GB of disks. That's only 2fps. Think about that at 20fps.

Thanks, I guess I need to experiment a bit with framerate to get this figured out. I have plenty of disk space to use for recording, so that's no issue. One of my concerns is that adding a monitor increases the load significantly on the server (even when I'm not viewing that monitor). Is this normal? Why would the load increase in this situation?

[Linwood]

One of my concerns is that adding a monitor increases the load significantly on the server (even when I'm not viewing that monitor). Is this normal? Why would the load increase in this situation?

I guess it depends on what you are doing when you add it, e.g. if it is set to record or to do motion detection, it is still having to process the data. The larger resolution images are decoded and converted to jpg's for each frame, so there is a relatively hefty amount of processing done just to handle that stream.

Is that what you mean?

The viewing, depending on what type, can actually be relatively mild in comparison. For example viewing it live it grabs a frame and sends it to your client but it tends to be self limiting, as the streamed jpg's cannot usually go all that fast, not like the 25fps nearly usually.

For good or ill, low frame rate will become your friend if dealing with high resolution. At least until (unless) zoneminder starts to be able to take H.264 streams and write directly to disk. Works is going on for that I know, but I have no idea where it stands. Though even then if you are doing motion detection it has to decode.

[bmather9]

One of my concerns is that adding a monitor increases the load significantly on the server (even when I'm not viewing that monitor). Is this normal? Why would the load increase in this situation?

I guess it depends on what you are doing when you add it, e.g. if it is set to record or to do motion detection, it is still having to process the data. The larger resolution images are decoded and converted to jpg's for each frame, so there is a relatively hefty amount of processing done just to handle that stream.

Is that what you mean?

The viewing, depending on what type, can actually be relatively mild in comparison. For example viewing it live it grabs a frame and sends it to your client but it tends to be self limiting, as the streamed jpg's cannot usually go all that fast, not like the 25fps nearly usually.

For good or ill, low frame rate will become your friend if dealing with high resolution. At least until (unless) zoneminder starts to be able to take H.264 streams and write directly to disk. Works is going on for that I know, but I have no idea where it stands. Though even then if you are doing motion detection it has to decode.

What really concerns/confuses me is that I add 4 of my cameras to ZoneMinder with the function of 'Monitor'. So I'm pretty sure that means No motion detection processing, and no recording is going on? Even when I'm NOT viewing the streams from those cameras, the load increases significantly, just by adding the monitors to the list in ZoneMinder.

[Linwood]

What really concerns/confuses me is that I add 4 of my cameras to ZoneMinder with the function of 'Monitor'. So I'm pretty sure that means No motion detection processing, and no recording is going on? Even when I'm NOT viewing the streams from those cameras, the load increases significantly, just by adding the monitors to the list in ZoneMinder.

Ah, sorry, when you were saying "monitor" I was just thinking camera, not the function.

In monitor mode it appears zoneminder continues to capture and decode the video stream. My presumption (complete guess) is so that it is ready and able to provide a live feed if you request it, since getting the initial feed going takes a while (e.g. on my cameras I have I frames every 25, at 4fps, so it could take up to 6 seconds to get a correct image if it wasn't processing all the time).

I don't know for sure that is the reason, but I turned some of mine from recording to monitor, and the zmc process is still running.

[Linwood]

PS. If you want to leave it in there but not taking any load while you experiment, try disabled. Obviously not convenient if you periodically want to look at the camera.

[bbunge]

Even with the monitor process ZM is still processing streams from the cameras. Lowering the frame rate is a good start. Lowering the resolution is good if you can. Move PATH_SWAP to RAMDISK (/run/shm for Ubuntu 14.04) as memory is a whole lot faster than writing to a drive!

I have 17 IP cameras most running 5 FPS @ 320x240 on an i5 with 8 GIG RAM. Right now with all cameras on mocord and skip frame set to 1 the "load" is at 0.60. When there are several folks viewing Montage vies the load can go up to 3.50. I am purging events at 30 days and am using less than 80% of 1.3 TB drive space.

bb

[bmather9]

PS. If you want to leave it in there but not taking any load while you experiment, try disabled. Obviously not convenient if you periodically want to look at the camera.

Disabling appears to have no effect on the load, which is also strange. Stopping ZoneMinder does reduce load significantly.

[Linwood]

PS. If you want to leave it in there but not taking any load while you experiment, try disabled. Obviously not convenient if you periodically want to look at the camera.

Disabling appears to have no effect on the load, which is also strange. Stopping ZoneMinder does reduce load significantly.

Well, I could come up with a good explanation for why monitor did not stop capture, but I have no idea why disabled does not stop it -- I just tried and zmc is still running. Strange.

Though all this is relatively moot for using it for real, since you would not leave it disabled.

[bmather9]

PS. If you want to leave it in there but not taking any load while you experiment, try disabled. Obviously not convenient if you periodically want to look at the camera.

Disabling appears to have no effect on the load, which is also strange. Stopping ZoneMinder does reduce load significantly.

Well, I could come up with a good explanation for why monitor did not stop capture, but I have no idea why disabled does not stop it -- I just tried and zmc is still running. Strange.

Though all this is relatively moot for using it for real, since you would not leave it disabled.

I stepped down to 1 camera so I can experiment. I spent about 15 mins recording and the load only reached about 0.6. After stopping the recording and switching the camera back to 'Monitor' my load jumped to 2.0 without changing anything else. Any idea what would cause this? I checked with HTOP and sure enough, it's zmc that's using the CPU.

In addition, where does zoneminder store recorded video? Or JPEG images as I hear? I checked the events folder and the images folder but there's nothing there? Although I have 3 events from my continuous recording showing up in zoneminder. You would think in software meant for cameras, to record video, that there would be some obvious name for the storage location and settings dealing with recorded video. I've googled and searched wiki for the past 30 mins but have come up empty handed.

[bmather9]

Ok, so I finally found where the images are stored: /var/cache/zoneminder/events

Unfortunately there is also a /usr/share/zoneminder/events but nothing exists here...

[bmather9]

Even lowering to 1 fps still sends the load well over 2.0 (dual core). That's still at the full 3MP resolution so I may try lowering that, but don't really want to. I also noticed that after stopping recording that nph-zms takes up 100% cpu for some time...is this because it's finishing up recording frames that were backed up during the time the processor was too busy?

[Linwood]

I wonder if you need to look at load over longer intervals? I cannot explain what you saw exactly.

zma is the analysis tool that does motion detection, zmc is the capture component, and zms (which is also nph_zms) is the outbound streaming component that let's you view.

zms (in my opinion) is a bit of a strange beast, as it pushes a stream of (usually) jpg images to your browser. Because it's a push, the termination process (again, to me) never seems clean, you usually get some log messages of errors of sockets failing or other messages. It is possible you are seeing a delay in that process ending as you stop watching.

zmc on the other hand is usually pretty stready, but zoneminder seems to restart the image handling when you update the camera with new settings, and it takes a bit for it to warm up (I think there's actually an option to specify how long in frames). You may be seeing that.

There are also other processes that kick off (and I have no idea in what program name) at times for audits (reconcile database vs file system), filter processing (if you have any, and maybe it wakes up to check), event completion and starting new ones (which may involve dumping some accumulation of frames to disk). I never "see" these events having a load impact, but I never have looked in isolation or carefully, so it is possible they are showing up.

There's also pre- and post-alarm activity that occurs (e.g. it accumulates a rolling buffer in case it gets an alarm and you told it to write pre-alarm frames). At very low frame rates and when you have these at the defaults (25 I think) this can result in delayed activity for several seconds of frames (but should not be several seconds of processing).

I am looking at a system right now with seven 3mp cameras (2048 x 1536) running at 4 fps, and saving images at 2fps. The load is showing as an average of 2. This is on an old (about 5-6 years) desktop Q6600 intel processor which has 4 cores, but hyperthreading shows as 8, so this is 1/4 busy. What I see is 4 zmc processes and 4 zma processes always at/near the top with zmc taking most of the horsepower (in fact accumulated CPU is about twice on zmc vs zma). mysqld pops up and down in bursts but is accumulatively a bit more than the zmc processes individually (I.e. maybe overall 10% of load). When zms runs it rarely reaches the top for me.

My bigger performance issue was memory; it was 6G and difficult to expand due to the motherboard (I have to replace it all, and can only go to 12 even then). With 7 cameras the shared memory is 77% in use and that's with 4G of the 6G allocated to it, and you clearly do not want shared memory swapping to disk.

These are all observations, not knowledge of the code, take with a grain of salt. My one recommendation is lengthen your observations and averages -- you will be more concerned about the steady state, including watching. Zoneminder tends to be fairly graceful in handling short bursts of activity (but not of memory exchaustion!), dropping some frames if needed to catch up. These short term fluctuations tend not to scale, i.e. what you see with one camera will not be six times as bad as with six for the short term events, it is the overall load from zmc, zma, and (only if you keep viewers up all the time) zms that tend to scale up linearly. And keep an eye on memory -- big cameras use a lot of it, notably whatever goes for shared memory on your server (mine is called /dev/shm in the df display). That also tends to scale linearly with number and size of cameras, AND you need some extra for viewing, etc.