Is ZoneMinder scalable?

[netwrkr]

Greetings:

Long time ZM user. I figured I would provide some feedback on a ZM setup we've been using the last few years.

Running ZM v1.24.4

Server Setup:
Dell PowerEdge R610
CPU: 2 x Intel Xeon E5530 @ 2.40Ghz
OS: CentOS 5.6
Mem: 16GB
Disk: (6) 146GB, 15K RPM SAS 6Gbps 2.5in
RAID Controller: PERC H700, 512MB Cache (RAID 10)

Cameras (26 total):
(2) AXIS 221
(5) AXIS 216FD
(1) AXIS M3014
(16) AXIS P3343
(2) AXIS P3346

Misc Notes

All cameras stream between ~27 and 30 FPS.
All cameras configured with 0 - 10% compression
All cameras capturing @ 320x240 (ZM setting)
A handful of the cameras are streaming via MPEG, the rest MJPG (the newer AXIS cameras only support H.264 which ZM does not yet support)
ZM configured for "Function" "Modect" on all cameras

System Load: Averages out to be ~3.60 over a 24hr period. Typically around 4.5 throughout the day.
IOWait: Averages 0.11% over a 24 hour period. Only above 1.00 when backups are running, overnight.
Bandwidth: The server receives ~140Mbps 24x7, from the various cameras. (Yes, all cameras are local to the server.)
Cameras average between 1Mbps and 9.75Mbps (I attribute this to using MPEG vs MJPG)

Based on ~390GB of usable disk space, and keeping disk space usage at or below 90%, system is able to retain about 4 - 4 1/2 months of videos before being filtered out.

In summary, is ZM scalable? My conclusion is absolutely. I think once ZM starts supporting H.264 the load will drop considerably, as it would if I reduced the FPS or increased the compression on the cameras.

I think I've covered the high points. If you have any questions, please feel free to ask.

[FeraTechInc]

That's quite the setup you have.

We were using iSpy previously with WindowsXP and 17 cameras. The system was very unstable. We switched to Zoneminder and never looked back. I must say it scales very well.

Are the Axis cameras you use POE? We have been doing these system for a while and we have found for larger installations POE cameras are the best to use. No more issues with needing power and Cat5e is very easy to run.

[netwrkr]

Yeah, we are using all POE for the cameras. Makes installation quite simple. I've done installations of both IP and analog cameras. Nothing beats the ease of install or the quality of picture you get with the IP cams. Axis may not be the highest end around, but for an office environment, with decent lighting, they produce a very high quality image.

[Flasheart]

I have similar camera numbers on a single cpu server, but nowhere near the fps you're achieving.

Scaleable - well, in theory, yes and no from what I understand. This /may/ be wrong, it's only my theory and I may have overlooked something.

Putting the sql onto a server of its own is a start. That will improve things a lot. That would scale up to a point, and mysql can be scaled indefinitely itself - instead of a single server it could be a cluster. There's quite a bit of documentation about doing that, and any reasonable dba I imagine will know the tricks.

You could then have multiple capture servers feeding that sql database remotely with data. As many as you want.

The images - that's a little trickier. Might be done by having a central san/nas mounted as zm's events dir by all capture servers and the database server. As each camera will be allocated a sub directory for itself, there shouldn't be any file locking problems.

For ongoing maint - one server, may not even be any of the above - can run zoneminder without any cameras and call zmaudit with a backgrounded task to autoexpire things.

Would be an interesting project if you do go ahead, please document it, love to know how you get it working!

[netwrkr]

Short of a massive configuration, I don't see the DB as being a limiting factor. As far as I can tell the DB only tracks limited information about the events. For example, my DB with 4 1/2 months of data is only 2GB is size. IMO the real work load is image analysis (CPU) and storage of video (IO). Depending on the frame rate and capture area I could easily see ~5 streams per Hyper-Threaded core.

It really boils down to how the system is setup with increased resolution and frame-rates being the main trade-offs.

Could a single ZM setup handle a "Las Vegas" casino? Not a chance. Even the smaller casinos have ~600+ cameras. Will a modest ZM setup scale past 30 cameras given reasonable frame rates and resolution? Absolutely.

When you consider the target audience for ZM - I'm guessing its in the 1 - 8ish camera range - it does that an plenty more quite well.

[Flasheart]

netwrkr - mysql does take quite a lot of resources on a busy system, it's the heaviest single process (zm being split into many). As you approach full system load the amount of slow queries starts ramping up quickly. I know from other db stuff personally that shifting the db onto another machine can make a massive improvement in performance.

I can't /see/ any reason why zm couldn't scale indefinitely given proper resources. It's built around technology that /has/ been scaled in other uses.

More the question is - would you? Your casino example is a good one, and there may be legal requirements for software certification that zm doesn't have in that sort of scenario. Also, some commercial software may be designed to scale from the start and with time always being a pressing concern in business, it may actually work out better to throw a bunch of money at a commercial system that someone else is already using successfully - plus they get the support when they need it.

[kross]

There's a slight problem in that the auditor and filter programs will get run on each host. I think they expect to have exclusive access to the system. So you would need to do something to ensure they are only run on a single host. Other than that, I don't see why you couldn't scale by adding more hosts.

[netwrkr]

@Flasheart - I can see where mysql can slow down quite a bit on a busy system. I've mitigated this to some extent by playing with the various SQL settings - the typical sort of things - increasing buffers, table caching etc. I am not a DBA but...thinking out loud, I would think multiple techniques could be implemented to reduce the load MySQL presents while using the MyISAM storage engine. Possibly looking at InnoDB which seems to do a better job of caching table data rather than just the keys - at the expense of more memory usage. I do know most of my slow queries is when MySQL has to examine ~37M rows which should be very fast except when it is also trying to do a bunch of other things to support ZM, concurrently. Again, I am not a DBA but 37M rows to examine seems quite excessive if indices are used properly.