Zoneminder processes being killed by OOM and OOMD on Fedora

[chue]

Hello, I recently installed Zoneminder on Fedora. I was having some issues with Zoneminder being killed by the system OOMD and OOM processes. I changed my settings for both and so far all seems ok. I'm hoping that what I post here can help someone. Or perhaps the powers that be can suggest better settings than I have used.

Install details:

Fedora 34 (server edition, very minimal install)
Zoneminder 1.36.12
Number of cameras: 2
fcgiwrap daemon procs: 6
Zoneminder Machine RAM: 8 GB

Files added/updated:

/etc/systemd/oomd.conf

Code: Select all

[OOM]
SwapUsedLimit=99%
DefaultMemoryPressureLimit=100%
DefaultMemoryPressureDurationSec=30s

/etc/sysctl.d/00-reserve-ratio.conf

Code: Select all

vm.lowmem_reserve_ratio = 128 128 32

I tried to update the OOMD configuration (oomd.conf) so that OOMD would not kill any processes. This seemed to work, but then to my surprise the old OOM process kicked in and would kill my Zoneminder processes.

To prevent OOM from doing this, I followed the advice from this link: https://blog.kamens.us/2016/11/10/linux ... of-memory/

The above link suggests changing the defaults from "256 256 32" to "128 128 32", which is what I did in the file 00-reserve-ratio.conf.

As I said, so far Zoneminder seems to be running well. I'll post an update here if that changes.

[chue]

Forgot to mention Zoneminder is running under nginx, and not Apache. It's probably not relevant, but thought I'd include this.

[chue]

Spoke too soon... Zoneminder has been killed. I will keep on playing with the values for the reserve ratio and see what happens.

Here's the service log, in case anyone is interested:

Code: Select all

Dec 30 16:53:03 zmstats[868]: INF [Stats Daemon exiting]
Dec 30 16:53:03 zmdc[805]: INF ['zmc -m 1' exited normally]
Dec 30 16:53:03 zmdc[805]: INF ['zmwatch.pl' exited normally]
Dec 30 16:53:03 zmdc[805]: INF ['zmstats.pl' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmfilter.pl --filter_id=1 --daemon' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmfilter.pl --filter_id=2 --daemon' exited normally]
Dec 30 16:53:04 zmdc[805]: INF ['zmc -m 2' exited normally]
Dec 30 16:53:04 zmdc[805]: INF [Server shutdown at 21/12/30 16:53:04]
Dec 30 16:53:04 systemd[1]: zoneminder.service: Failed with result 'oom-kill'.

[chue]

Adjusting the reserve ratio did not work. I kept halving the numbers until I reached the minimum. In other words, I tried these values for vm.lowmem_reserve_ratio:

Code: Select all

128 128 32
64 64 32
32 32 32
16 16 32
8 8 32
4 4 32
2 2 32
1 1 32

However, after each adjustment OOM always kicked in and killed ZM.

I then did a fresh install on Fedora 35 (not 34), but included 8 GB of old style disk swap space. By default Fedora 34 (?) and later do not include this. They include a compressed swap (zram) that exists in RAM only.

In this install I did not do anything to adjust OOMD or OOM.

In any case, Zoneminder seems much happier with the included on disk swap. It ran for 24 hours and wasn't killed, whereas without the disk swap ZM would be killed within 12 hours.

As an extra test, I removed the swap from the working install. Immediately, ZM was killed by OOMD.

For now, I'll add the swap back in and run with it. Hopefully this is solved.

[chue]

Not solved. ZM keeps getting killed. It looks like ZM is hanging onto too much memory. If there's anything else I can post here that can be of help, let me know. Here's a "top" of the system after running for a while:

f35sec3-top-w-virt-mem.JPG (97.86 KiB) Viewed 79545 times

[Magic919]

Try disable saving JPEGs if you haven't already.

[chue]

Thanks. I will try that and report back. "Save JPEGs" was set to "Analysis Images Only (if available)".

[chue]

Just to add some more information, here are the settings for one of the cameras. There appears to be a limit, so will add more settings in another post.

mo-1-settings-general.JPG (61.32 KiB) Viewed 79514 times

mo-1-settings-source.JPG (38.26 KiB) Viewed 79514 times

mo-1-settings-storage.JPG (32.24 KiB) Viewed 79514 times

[chue]

Settings for buffers:

mo-1-settings-buffers.JPG (30.56 KiB) Viewed 79514 times

[chue]

Try disable saving JPEGs if you haven't already.

ZM got killed by OOM after 3 days (?) after disabling JPEGs. I will note that I was running with OOMD off (see oomd.conf in my first post).

Here are some "top"s over time. I only got 2 days worth. ZM was killed about 12 hours after the last image.

Day 1:

f35sec3-top-no-jpg.JPG (88.2 KiB) Viewed 79474 times

Day 2:

f35sec3-top-no-jpg-day-2.JPG (77.56 KiB) Viewed 79474 times

Day 2 (some hours after above):

f35sec3-top-no-jpg-day-2b.JPG (66.24 KiB) Viewed 79474 times

[chue]

Also, for some reason I could no longer see any live views (e.g. montage or camera view). I need to investigate if disabling JPEGs caused this.

I will add that if ZM crashes after 3 days, I can work around this with nightly reboots. Still probably needs some investigation by the devs at some point though.

[chue]

Turning off Save JPGs does disable live view, which is a dealbreaker. I don't know what to do at this point, perhaps I'll try a different distribution perhaps CentOS or Ubuntu. Or perhaps install an older version of ZoneMinder (1.34).

[Magic919]

My live views are not affected, so for me it's a useful way to cut memory use.

[chue]

I installed an older version of ZM (1.34.26-1) on Fedora 34. This was installed without on disk swap, so with only compressed swap in RAM.

This older version of ZM was unavailable on Fedora 35, otherwise I would have gone with the newer Fedora.

The memory usage has been low and stable over time - memory usage does not grow. I also noticed that on this version of ZM, when something moves on camera, the live views stay responsive. The newer version live views would freeze for 20 or more seconds. So for now this is what I will go with.

Here is a "top" after a few days:

f34sec4-day3.JPG (75.66 KiB) Viewed 79281 times

[chue]

Memory usage still quite stable after 5+ days

f34sec4-day5.JPG (79.49 KiB) Viewed 79180 times