High CPU load after upgrade 1.34 -> 1.36

[migo]

Hi, my server load rises from 14 to 25 after upgrade from Debian oldstable to stable. I've made no configuration changes. Server is working fine with ZM for ages, it is 2xIntel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz with 128GB RAM. Here are some graphs documenting the change:

Screenshot from 2024-09-06 07-47-44.png (65.69 KiB) Viewed 1776 times

Screenshot from 2024-09-06 07-48-04.png (81.18 KiB) Viewed 1776 times

Screenshot from 2024-09-06 09-25-22.png (58.29 KiB) Viewed 1776 times

System is running 71 modect monitors. Can anybody give me a hint where should I search please.

Thank you in advance!
Milan

[stafflabour]

Hi, my server load rises from 14 to 25 after upgrade from Debian oldstable to stable. I've made no configuration changes. Server is working fine with ZM for ages, it is 2xIntel(R) Xeon(R) CPU E5-2640 0 @ 2coreball.50GHz with 128GB RAM. Here are some graphs documenting the change:

Screenshot from 2024-09-06 07-47-44.png
Screenshot from 2024-09-06 07-48-04.png
Screenshot from 2024-09-06 09-25-22.png

System is running 71 modect monitors. Can anybody give me a hint where should I search please.

Thank you in advance!
Milan

I would like to understand why the gap value appears!

[iconnor]

Please read the release notes.

You are likely using double the ram due to ImageBuffers not being updated.

[migo]

Hi, my server load rises from 14 to 25 after upgrade from Debian oldstable to stable. I've made no configuration changes. Server is working fine with ZM for ages, it is 2xIntel(R) Xeon(R) CPU E5-2640 0 @ 2coreball.50GHz with 128GB RAM. Here are some graphs documenting the change:

Screenshot from 2024-09-06 07-47-44.png
Screenshot from 2024-09-06 07-48-04.png
Screenshot from 2024-09-06 09-25-22.png

System is running 71 modect monitors. Can anybody give me a hint where should I search please.

Thank you in advance!
Milan

I would like to understand why the gap value appears!

Which one?

[migo]

Please read the release notes.

You are likely using double the ram due to ImageBuffers not being updated.

Hi, thank you for your reply. Do you mean this:

When editing buffer settings, ensure that MaxImageBuffers > PreEventCount.

Thank you!

[iconnor]

No. In 1.34 ImageBuffers had to be large... Now it can be something more like 3. During the upgrade the value should have been copied to MaxImageBuffers and set to 3, but I've seen lots where that didn't happen, and so, both are large and the system is unnecessarily using double the ram.

[migo]

Weekend looked promising, system load was around 16, but today when there was lot of movement under cameras, system load peaked at over 60.

Screenshot from 2024-09-09 15-29-04.png (44.18 KiB) Viewed 1642 times

There is lot of free usable ram:

Screenshot from 2024-09-09 15-30-46.png (74.59 KiB) Viewed 1642 times

I'm trying to do zoneminder downgrade now to 1.34.

[migo]

No. In 1.34 ImageBuffers had to be large... Now it can be something more like 3. During the upgrade the value should have been copied to MaxImageBuffers and set to 3, but I've seen lots where that didn't happen, and so, both are large and the system is unnecessarily using double the ram.

Thank you for clarification. Initially there were large values from v1.34, I've adjusted all to 0 - automatic setting for Maximum Image Buffer Size, right after upgrade... Image Buffer Size was set to 3.

[migo]

I was able to downgrade Zoneminder to 1.34 using Deb11 package and I also needed to install and use php7.4 for it. I've dropped 1.36 database and imported backup from 1.34. Everything else is the same like was with 1.36. I've let it settle a bit to not make false conclosions. Now is Zonemnider working like it should. I've downgraded Zonemnider at 11 AM yesterday, it can be seen on graphs below.

Screenshot from 2024-09-10 08-17-31.png (47.47 KiB) Viewed 1590 times

I don't think that 1.36 was using double the RAM, please look at memory usage graph:

Screenshot from 2024-09-10 08-16-36.png (86.04 KiB) Viewed 1590 times

Actually 1.36 was using less RAM than 1.34 but not using shmem at all.

Interesting is that 1.36 was making a lot more context switches:

Screenshot from 2024-09-10 08-18-02.png (69.98 KiB) Viewed 1590 times

and VM stat looks more jerky. DB usage pattern is different too, but this comes from changes you have made in code, no table locking etc.

I've monitored system load with atop, and it was CPU bound only. So no network or IO problems were detected. With this test I can rule out HW problem and Debian12 influence on Zoneminder's performance.

Any ideas? Thank you very much!
Milan

[migo]

Here is CPU usage graph:

Screenshot from 2024-09-10 08-28-12.png (59.14 KiB) Viewed 1590 times

and VM stat graph:

Screenshot from 2024-09-10 08-28-31.png (71.9 KiB) Viewed 1590 times

[MechanicMike]

Hey Milan, that sounds frustrating, man. When you're facing a sudden spike in CPU load after an upgrade without tweaking settings, it usually boils down to how the new version is utilizing resources differently. Since you got a beefy setup, those 2xIntel Xeon CPUs and 128GB RAM should handle ZM with ease. But if it's spiking unexpectedly, maybe there's a new feature in 1.36 that's more resource-intensive. I'd recommend checking out ZM's release notes for the new version to see if any changes might be causing this. You might also want to monitor specific processes to see which one is causing the strain. Good luck, bro!

[migo]

Thank you. The biggest change to me was merge of zma and zmc processes according to change log, but developers should have better insight. Maybe it is something with encoding/sending data to web browser.

Here are more details about our system: We have 5 computers running 5-6 cameras onscreen non-stop within Firefox web browser. 40% cameras are 640x480 resolution, 40% HD and rest FullHD all are 25FPS. All cameras are streaming h264 passthrough and not saving jpegs. All cameras are modect with Analysis FPS set to 5.

After downgrade to 1.34 is all fine all the week:

Screenshot from 2024-09-17 08-52-09.png (40.92 KiB) Viewed 1303 times

[migo]

In addition, here is network usage graph, incoming traffic from cameras is 200Mbit/s and is +- constant, outgoing traffic is to the 5 PCs used by surveillance personnel and varies upon stuck live video in browser windows on those computers. The CPU load is not dependent on count of connected clients with Zoneminder 1.34 as can be seen here.