HW crisis after upgrade to ZM 1.36.xxxx

[webdriver]

First post but like to discuss something happened recently to my ZM system.

My old system (1.34.xx) run on a Lenovo Tiny PC with a dual core Celeron + 8GB memory and a 7200HDD, can handle my 6 cameras (2 poe, 4 wireless).
Then one day I decide to upgrade to 1.36.xxxx as it always prompt on the dialog -- ok -- I didn't try upgrade on spot instead I re-install Ubuntu LTS 22 + ZM ... everything looks good on the start, and it can handle everything as my 1.34 system. But one day later, there is no way I can access the web GUI nor hard to SSH onto the system as IT's RUNNING so slow. I knew I have overly used my dual core/8G so I thought alright it's time to move to a newer/better HW. By the time I had another Dell Micro PC (yes I like these tiny form PC so had different brands in my house) with a 7th i3 + 12G memory, I decide to decommission Lenovo dual core to this one. Everything looks much better after install Ubuntu + ZM on the new Dell PC, but after some days the same thing happens -- it's not CRASH but just extremely slow to access ZM GUI and also SSH became slow. Finally I got onto the system and found out my 12GB memory only left 100MB <--- I don't know why 1.36 became so hungry on memory, and looks like I am not the first one reporting ...

[webdriver]

Things get worse after my first post.
I replace my memory from 12GB to 16GB and run for a whole night. The memory usage from 7GB (start service) and now is 12GB used + some 3G buffering/caching and only leave 700MB free. Definitely something wrong with new version of serious memory leaks.

Any good suggestion on a cron job to stop/restart the zm service? (before the problem got fixed )

[webdriver]

Capture.JPG (51.01 KiB) Viewed 2782 times

[Magic919]

Daily restart might keep it under control. That’s not to say that buffer tuning, maybe some other tweaks wouldn’t help. Plenty of threads on the topic here.

[webdriver]

Daily restart might keep it under control. That’s not to say that buffer tuning, maybe some other tweaks wouldn’t help. Plenty of threads on the topic here.

Yep, digging with "memory leak" got me some early posts

[mikb]

Things get worse after my first post.
I replace my memory from 12GB to 16GB and run for a whole night. The memory usage from 7GB (start service) and now is 12GB used + some 3G buffering/caching and only leave 700MB free. Definitely something wrong with new version of serious memory leaks.

Any good suggestion on a cron job to stop/restart the zm service? (before the problem got fixed )

On the "top" you posted ...

Or to look it another way, 700MB wasted doing nothing ... outrageous!

This isn't Windows. This isn't DOS. This is Linux. It is normal for all of your "unused" memory to get eaten up. People used to lesser operating systems panic and think that's a fault.

If you're not using it, you won't miss it, and the O/S can make better use of it than you are -- that's why 3G is being stolen for "buffering and caching". Look at your swap -- 2G total, 2G free. This system isn't short of memory, yet. It's keeping files in memory to speed up access.

Would you rather it gave you 3.7G of free memory but read everything from disk, every time? Choose carefully.

As you start to actually USE (or leak!) memory, the "buffering/caching" amount will go down, as the OS gives you back memory for use. It's good to have some free so that things can allocate memory without first having to de-allocate something else, but that's what the 700Mb is for.

As you really start to USE memory, the 2G swap will get eaten into as things are pushed out to disk. That's slower. Eventually, things will run slower as things swap in and out, and the OOM killer will kick in and start terminating things it perceives to be screwing up the memory usage. It's that, or crash.

[webdriver]

Thank you for the input -- I just capture something running after merely 10 hours.
The worst case I saw on the system only have less than 100Mb from "top" report and what alert me of that when access the Web GUI or SSH is comparably slower than other time -- note: my system/data all stays on HDD not SSD, so when swap happening, I can imagine the slowness.

The problem is when on 1.34 system my celeron CPU + 8GB + HDD did well but move to 1.36 on a system with better CPU, more RAM could slow down make a lot of non-sense.

[iconnor]

I suspect that you need to edit your buffer settings. WE changed a lot in 1.36. Set your ImageBuffers to 3, and set your MaxImageBuffers to whatever you used to have in ImageBuffers.

I suspect you are using double the ram than you should be.

Otherwise, post pics of your monitor settings. Maybe give us some info to go on.

[keithp]

Just to chime in on this thread to help others being hit with memory problems...

1) Currently trying to find a "good" setting for MaxImageBuffers is likely going to drive you nutty, especially if you are using a mode that does analytics. That is what seems to blow the system up overtime. My solution was to set it to a number that is high enough to no log an error and set any cams that are using "Mocord" (and possibly "Modect") to "Record". I didn't play with the "Analysis Enabled" flag because I figured at this point, having a complete recording without alarm tags in the replay is better than nothing at all or dealing with camera zmc processes being killed by the oom constantly.

2) I had "Warmup Frames" and "Stream Replay Image Buffer" at non-zero figures. They both can be set to 0. In particular, "Stream Replay Image Buffer" writes jpeg to storage to pause and rewind live view. If you don't do that, you don't need it. Thanks to iConner for these recommendations, I think this helped a lot in my case.

I did these things on 8/11 and my system has been stable (5 days and counting). No lockups at all and I'm only seeing two cases off the oom killing a zmc process. I have gotten "Unable to stream" messages every so often on a cam while I was watching on the Montage but it doesn't seem to time out like that for long anymore.

[webdriver]

Update my progress:
I was reading some of the discussion about "watch/detect on low res stream" but "record high res video" which attracted my attention.
I followed the suggestion (zoneminder way) and changed 3 of my camera doing the trick -- it worked well so far and interesting enough my overall memory usage dropped -- now I had around 4-5G out of 16GB total

Capt333ure.JPG (56.99 KiB) Viewed 2708 times

Although it's fluctuated up & down but some hours past it won't eat up all memory like before. I will keep monitoring for some days.

----------------------------------------------------------

[webdriver]

I suspect that you need to edit your buffer settings. WE changed a lot in 1.36. Set your ImageBuffers to 3, and set your MaxImageBuffers to whatever you used to have in ImageBuffers.

I suspect you are using double the ram than you should be.

Otherwise, post pics of your monitor settings. Maybe give us some info to go on.

Thanks for the recommendation, I actually didn't pay much attention to "buffering" settings, every single camera I added/edit use the default settings other than 3 tabs "General","Source" and "Storage".

Capxfsdfture.JPG (30.34 KiB) Viewed 2707 times

I will try play with it and see any improvement on detection/recording result.