Performance Issues - What am I doing wrong?

[cherokee_pilot]

Hello,

I am trying out zoneminder 1.36.33 and am enjoying it so far. I've seen numerous threads on optimizing performance and how it can really scale to handle a lot of cameras. I've seen the Dummies guide in the wiki. And I believe I've configured my new ZM instance to follow the recommended best practices - more details on my setup below.

I'm running zoneminder in a virtualized Ubuntu Server 22.04 on an older chassis with 10 vCPUs and 32GB of RAM (these are zoneminder's resources, not the server chassis itself). The underlying hardware is dual Xeon E5-2670 chips and 128GB DDR3. There are a couple of other very low volume/load VMs running on the same server and I don't see any resource constraints from the hypervisor perspective. I am monitoring the hardware utilization of the ZoneMinder VM itself with node exporter (and some sanity checks with htop), and the CPU is consistently at 80%+, and memory is at about 23GB used.

I have 10 cameras currently added. All are using RTSP for transport. Eight of them are 4MP cams sending 2688x1520. Two of them are 3MP sending 2048x1536. All 10 are in Nodect mode, and I've unchecked "Decoding Enabled" in each one of these main streams.

For motion detection, I have VGA (640x480) substreams setup for the eight 4MP cams (haven't figured out the substream URL for the 3MP cams, yet).

All eight substreams are in Modect mode and Analysis FPS is set to 5 FPS. The substreams are configured (at the IP cam itself) to 15 FPS. Each substream has one zone that I've set to the appropriate area (typically between 50-80% of the entire viewpoint)

So, is this a normal/expected load for these cameras? I really feel that I setup something wrong, because I'm also using these same exact cameras with QNAP's Surveillance Station, with motion detection, on a NAS with much less capable hardware (Intel i3-7100 and 16GB RAM), yet it's handling these cameras without breaking a sweat (<30% CPU usage and < 8GB RAM usage on the entire box, and I have a couple other VMs and containers running on this one, as well, on top of all the NAS-related stuff it's already doing!)

My only goals are to maintain real time live-view functionality, and have a timeline with motion events that can be reviewed. I'd like to be able to throw some Raspberry Pi based IP cams on and setup email notifications, but I've not yet looked into either of these.

Thanks for a fantastic FOSS project and any tips you can provide!

[burger]

It's recommended to use dedicated hardware not VMs for Zoneminder. For the streams, you can do the following:

• use full res stream
• set camera in zm to passthrough (not decode) (must be h264, not h265).
• set resolution to be lower than the actual stream (e.g. if you have a 2K stream, set it to 320x240 or 640x480.
• set analysis fps to 2
• mode can be modect or mocord. I prefer modect with some exceptions.

By doing this you will get a low res live view and analysis, but the recorded videos will be full res when watched. This is the easiest way to setup ZM. You can also use linked monitors or have multiple streams, but neither of these are worth the trouble.

[cherokee_pilot]

Hello, thanks for the recommendations! While I can't convert my server chassis to run ZM on bare metal, I do have another less capable box that I've setup to give this a shot. It's an Intel NUC with an Intel i5-7300U (basically, a mobile/laptop grade CPU ), but it has 32GB of DDR4, so I thought maybe it'd have a fair chance.

I've installed Ubuntu server directly onto the NUC and configured all the monitors to 320x240 with camera passthrough.
Analysis FPS is set to 2 for each stream, and just like before, I have only 8/10 cameras setup in Modect mode (the other two are Nodect).

I've also gone through and fine-tuned the zone for each stream (same as before - one fairly large zone on each stream, taking up 50-70+%) of the view.

In the logs, I see constant mentions of "Decoding is not keeping up. We are 167 seconds behind capture.". I've looked at other threads here on the forum and the (quite obvious) answer is the same - the CPU can't handle the load. Not surprised with this mobile grade dual core CPU.

I think my only option at this point would be to reduce the resolution of the stream at the IP cam itself so that there are less pixels to decode. However, I'm not too interested in going down that path since I have a system that can handle the full resolution as is, so I think this concludes my foray into ZM. Maybe in the future I will get my hands on a used server with more modern processors and revisit ZM.

Hopefully this thread supports others in making hardware decisions and tuning their boxes.

PS - Here's the poor little NUC struggling to keep up:

[dougmccrary]

From a recent experience, check the setup (aka bios). I just had a performance problem with a new to me machine that had a couple of settings in default, and changing them to "performance" or "priority" settings made a huge difference in zm performance.
I've not used a NUC, so can't be specific.

[iconnor]

At the end of the day, you are still decoding a high res stream. That's a lot of work.

If you want to use no cpu, then turn off decoding or use decoding=Keyframes

[Boremon]

At the end of the day, you are still decoding a high res stream. That's a lot of work.
geometry dash subzero
If you want to use no cpu, then turn off decoding or use decoding=Keyframes

Thanks for your solution. I tried turning off decoding and it really helped me.

[cherokee_pilot]

At the end of the day, you are still decoding a high res stream. That's a lot of work.

If you want to use no cpu, then turn off decoding or use decoding=Keyframes

Completely understood - and thanks. Doesn't disabling encoding also disable live view functionality?

[iconnor]

Yes. The ondemand part means that when you go to view the stream, it will start decoding fulltime.

But you can setup janus or rtsp2web to do the livestream instead. (but only for h264 at this time).

[gbeyer]

[*] set camera in zm to passthrough (not decode) (must be h264, not h265).

I can find all of the other recommended settings but this one. Where can I find? Thx! [edit] Found it! Console / Source / Storage


Thanks for referring to the "Dummies" in the wiki, OP, most helpful stuff in there!

[gbeyer]

"set resolution to be lower than the actual stream (e.g. if you have a 2K stream, set it to 320x240 or 640x480."

My camera stream is 1920x1080, so I scaled down my resolution in Source to 640 x 480. But then I discovered I wasn't getting any more recordings.

I did a check of my zoneminder service, (systemctl status) found it was throwing errors:

Jul 23 14:36:58 zm-server zmc_m1[136215]: WAR [zmc_m1] [Monitor dimensions are 640x480 but camera is sending 1920x1080]

Upon restoring the 1920x1080 resolution, and restarting zoneminder, no errors and I'm getting recordings again. So how do I follow the advice given above?

[burger]

I think it has to be some fraction of the original stream. It must keep the same aspect ratio. I updated the wiki.

For 1920x1080 try 480x270.

Note that the message you are getting is a warning, not an error.