Large scale Zoneminder

[chipdewolfe]

This is pretty experimental...I've run Zoneminder with 10-20 camera setups, but never anything this big. I am having a very difficult time managing the load...none of my configuration changes seem to have any impact.

I have a new Zoneminder 1.25 setup on a CentOS 6.2 64-bit system. The machine has 4 Opteron 6276 CPUs, which each have 8 hyper-threaded cores that appear as 16 cores to the OS...in other words, 32 physical cores that appear as 64 cores. The system has 32GB of RAM. It also has a 22TB SAS storage array built on RAID 10 using 2TB Constellation drives. Machine is connected to a network with 1GB backbone and 100MB PoE to the cameras. The cameras and Zoneminder are on their own dedicated VLAN. I have the /dev/shm (tempfs) set to use 24GB, and it is operating with roughly 9GB free. I also have roughly 5GB of physical memory free and the system is not utilizing any of the configured swap space. I am using libjpeg-turbo. CPU utilization is running at 3-5%, CPU load is around 50. Iftop shows me that the network utilization is around 38Mb.

I have 58 monitors...all of them connected to an Axis P3304-V. They are all configured identically...a limit of 2 frames per second, 1280x720 resolution, JPEG compression of 60 (and I am pulling the JPG feed, not the MJPEG). The camera is sending images with the timestamp, so Zoneminder is configured not to. All monitors are on continuous record, so there's no analysis being done at the server level...simple store and index. Image Buffers set to 100 (I was getting Buffer overruns at lower values). I'm not seeing errors of any kind with the current config in the logs...it's just dropping frames (rather than 2fps, I'm getting .3 to .5 fps). I have been unable to reduce the load at all by (1) lowering the frame rate from 5fps to 2fps (on both the camera and Zoneminder), (2) increasing the camera's compression of the JPGs from 30 to 60, (3) reducing the resolution from 1280x800 to 1280x720, (4) removing the timestamping that Zoneminder was doing.

I can't throw any more CPU at it, RAM doesn't seem to be the bottleneck...any ideas?

TIA,

Chip

[henke]

Hi!

Interesting.

Does the speed increase if you switch the monitors to monitor function only?
Have you tried mpeg4?

[chipdewolfe]

Replying to my own post. I have Zoneminder running like a well-oiled machine now. The hunch that paid off was that xfs was the problem. Anytime that I did a process list, there were tons of xfs processes going on. I know xfs doesn't like lots of small files, but I went with it because it was the only file system built into a RHEL-derivative that could handle a 22TB partition (since RHEL still doesn't include the updated ext4 utilities that give access to larger partitions, ext4 was off the table when I built the server originally). I decided to format and rebuild the server once I was able to locate the necessary files to get greater than 16TB support in ext4. To get this support, I needed to update e2fsprogs, e2fsprogs-debuginfo, e2fsprogs-devel, e2fsprogs-static, e2fsprogs-libs, libcom_err, libcom_err-devel, libss, and libss-devel from yum(dot)aclub(dot)net/pub/linux/centos/6/umask/x86_64/. For stability's sake, I don't generally like to go away from the distro repos for system files, but, in this case, the distro repos weren't helping. I rebuilt the server exactly as before, except with my 22TB partition ext4 formatted. I recreated all of the monitors, still had them on 800x450 at 1fps with compression of 60. Started the ZoneMinder service and WOW! I had a sustained load of less than 0.10 with a peak of 1.47 doing continuous recording of all 58 cameras. So, I got brave and turned everything up to 1280x800 at compression of 30, still with 1fps. My load is now less than 0.20 recording all 58 cameras. Have yet to drop a single frame on any camera. I'm going to let it roll this way for a few days and see how things go. If all is well, I may turn up the fps a bit. I may even see if I can roll another building's security cameras onto this machine, which went from being highly overutilized yesterday to very underutilized today. Hopefully this helps someone else.

[chipdewolfe]

Hi!

Interesting.

Does the speed increase if you switch the monitors to monitor function only?
Have you tried mpeg4?

I hadn't tried MPEG4, but I had switched to monitors only on half the cameras, and it cut my load in half. I went from 52.00 to 26.00 lighting up only 29 cameras to record. In the end, I'm pretty sure it was XFS. It's the only thing I changed when I formatted and reloaded and all seems well now (see other post).

Thanks for the input!

[Flasheart]

Thanks for following up - that's interesting to know and I would not have guessed it would have made such a big difference. I use XFS on a couple of zoneminder servers because it doesn't have the 64k file-per-dir limit of ext4 and I wasn't using the deep file system setting in zm, and it works ok - BUT my servers they are 1-4 tb only.

Each of our buildings are geographically diverse and rural, so need their own local server rather than a central one.