ZoneMinder Forums

TL;DR: Zoneminder processes go out of control after a day or so of running cause huge load (upwards of 100). Where do start looking for the cause of this?

Setup: I am running 8 IP cameras at a remote location. The are ZModo ZP-IBH13-W. These cameras are set to broadcast at 720p 4fps due to bandwidth issues going over the VPN connection. There is an OpenVPN server at this location.

At my location I have a Dell Poweredge 2950 server running a couple of Virtual Machines. One of these virtual machines is running Ubuntu 16.04 w/ Zonemineder 1.29 (8gb of RAM, 4 cpu cores). I used this guide to setup Zoneminder: https://wiki.zoneminder.com/Ubuntu_Serv ... y_way.This VM is VPN'd into the remote network via the OpenVPN server. It is a bridged connection, so it appears in the same subnet as the cameras.

Zoneminder is setup to read streams from these 8 cameras with a RTP FFmpeg streams (found here: viewtopic.php?t=22573). I have all the cameras set to max out at 8 fps because when the VPN connection drops, FFMPEG tries to read in the streams at 200fps cause huge server load. I also have most of the feeds set to use black and white to reduce server load.

I bought zmNinja-pro to watch the cameras on my phone (https://play.google.com/store/apps/deta ... mninja_pro).

Everything works fine initially (recording, motion detect, web interface, mobile app). Server load stays around 1 when there is limited activity and jumps to 3-4 when there is more activity. This is find since I have 4 cpu cores provisioned to the VM Zoneminder is running on.

The issue: After about 24 hours of working correctly, something in Zoneminder goes haywire. The load jumps very high (upwards of 100) causing nothing to work excpet ssh on the VM. As soon as login and restart the Zoneminder service, load falls back to nomal levels and everything begins working again.

The questions: Has anyone had this issue or something similar? Where should I start looking to identify and correct this issue?

It is possible that 8 GIG may not be enough for your setup. Check the space available in tmpfs. Also run mysqltuner to see if MySQL parameters need to be adjusted.

See if you can identify which process is causing it, zmc, zma.

@bbunge: The memory usage never seems to climb over 2gb. While the rest ends up allocated in cache, it is never reported as used.
Had to restart again. Waiting to get a couple days of data for mysqltuner.

@SteveGilvarry: I will see what I can get out of it net time the load jumps.

@bbunge: (more specifically, here is a typical memory usage from `free -m`)
total used free shared buff/cache available
Mem: 7983 911 151 511 6920 6225
Swap: 8187 269 7918

Use "df" to check the tmpfs usage. tmpfs is also called ramdisk and is mounted and used as part of the file system. In Ubuntu 16.04 one tmpfs is /dev/shm Could be different for your OS Whatever it is you need to check the directory used by ZM PATH_MAP

@bbunge Here is the outtput of df while the VM is functioning normally:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 4068412 0 4068412 0% /dev
tmpfs 817524 8884 808640 2% /run
/dev/mapper/KCC--vg-root 2490835648 2128803936 235481492 91% /
tmpfs 4087600 495032 3592568 13% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 4087600 0 4087600 0% /sys/fs/cgroup
/dev/sda2 483946 103322 355639 23% /boot
tmpfs 100 0 100 0% /run/lxcfs/controllers
tmpfs 817524 0 817524 0% /run/user/1000

I could not ever ssh in the last time the VM acted up. I just restarted it. Is there program Is there a way to log the output of a program like htop for these instances?

VM went unresponsive again. I added this line:

* * * * * echo "=================================================" >> ~/cpu.txt; top -b -n 1 >> ~/cpu.txt;


to my crontab. Hopefully I can catch the state of the active processes before the machine goes unresponsive.

Here is the last top output at 2pm (which is apparently when it went unresponsive). I rebooted it around 10pm. CPU usage looks acceptable. Maybe its I/O bound?

So I switched from Record to Modect and the issue seemingly has gone away. I suspect the system ended up I/O bound when it went unresponsive. Since I do not need full time recording, this solution works for me.