Page 1 of 1

High CPU when camera is disconnected

Posted: Tue Jan 17, 2017 5:52 pm
by DoubleHP
Hello; got 6 cams on a weak server. reduced the frame rate to 3 inside each camera. My CPU usage is about 1/3 by night when activity is low around, and about half during day. I have found settings that work for me, most of the time.

3FPS is low rate, but, considering other factors ... does the job for me.

But if for any reason a camera is unreachable (cable disconnected, dead supply, maintainance), then the CPU load goes 70-80% for a single missing cam; and 100% for 3. In short, disconnect or break half of my cams, and the remaining ones also stop working due to system overload.

I have found that missing camera also produce a HUGE AMOUNT OF LOGS. Like hundreds MB per day.

How to reduce CPU usage, system load, and log lines when cam goes away ?

One possible scenario: I go on holidays for 1 week, first one cam goes off (dead psu), when I come back server is completely down due to disk full.

Re: High CPU when camera is disconnected

Posted: Tue Jan 17, 2017 6:33 pm
by rockedge
I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.

Re: High CPU when camera is disconnected

Posted: Tue Jan 17, 2017 8:43 pm
by DoubleHP
I don't have SHM issues; I monitor everything via Munin, and SHM use does not change significantly during these issues.

Which log file tells you that ?

# grep -nr "disk_free_space" /var/log/zm/
#

... nothing.

Re: High CPU when camera is disconnected

Posted: Wed Jan 18, 2017 12:21 am
by rockedge
that message came directly from the log setup as installed. Log was red and I clicked on it and that was in the log displayed in the window. First time I ever saw this message or error since version 1.26.5. In the options logging is set to write to the database.

Re: High CPU when camera is disconnected

Posted: Wed Jan 18, 2017 3:26 pm
by rockedge
turns out another log file, not related to ZM, had kept growing and loaded into RAM and eventually took up the available memory. I deleted the log and have not had the problem since. I will manage this log better and the problem looks solved.

Re: High CPU when camera is disconnected

Posted: Wed Jan 18, 2017 5:55 pm
by DoubleHP
Use Munin, and plugins loggrep and df, to get warnings about real causes weeks before this kind of trouble occurs.

Re: High CPU when camera is disconnected

Posted: Thu Jan 19, 2017 6:56 pm
by knight-of-ni
Workaround to high cpu usage when a camera drops offline is to put a frame rate slightly higher than the incoming stream into the Max FPS field.
For IP streams never place a value that is lower than the incoming frame rate in this field. This is described in the help text.

So in your case, put a 4 in the max fps field.

You can't reduce the amount of log entries going into the logs. The solution to this is to design a system where the cameras do no drop offline.
If however, you cameras go offline during a known time period, then you could make use of run states to disable the camera in zoneminder during those times.

Re: High CPU when camera is disconnected

Posted: Thu Jan 19, 2017 7:04 pm
by knight-of-ni
rockedge wrote:I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"

Re: High CPU when camera is disconnected

Posted: Thu Jan 19, 2017 7:15 pm
by rockedge
knnniggett wrote:This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
Thank you! That does seem to be exactly whats happening. I'll live with it until the next release.

Re: High CPU when camera is disconnected

Posted: Fri Jan 20, 2017 5:52 pm
by DoubleHP
YES, Maximum FPS had an impact on CPU usage. Setting this value reduced CPU usage even below the usage ratio I normally have when cams are working.

But logs generated by ZMA were almost the same. So, that variable affected only the CPU usage; not the log flood. About two lines per second, for the whole duration of issue, when disconnecting 3 cams. Or, one line per second per cam (depending how I read my stats). If I got out for one week, that can be a huge amount of disk space for a small server like a rPi.

Re: High CPU when camera is disconnected

Posted: Thu Jan 26, 2017 9:34 pm
by StephenOz
I had an issue were ZoneMinder was jumping to 100% CPU usage if the network interface was down - setting the maximum FPS solved this. Thanks for the info!

Re: High CPU when camera is disconnected

Posted: Fri Jan 27, 2017 8:01 am
by PacoLM
knnniggett wrote:
rockedge wrote:I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
I also noticed it. /run/shm keeps growing. In my case, the workaround is to restart Zoneminder from time to time.

Re: High CPU when camera is disconnected

Posted: Fri Jan 27, 2017 1:04 pm
by knight-of-ni
PacoLM wrote:
knnniggett wrote:
rockedge wrote:I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
I also noticed it. /run/shm keeps growing. In my case, the workaround is to restart Zoneminder from time to time.
This has been fixed in master:
https://github.com/ZoneMinder/ZoneMinder/pull/1717

Re: High CPU when camera is disconnected

Posted: Fri Jan 27, 2017 5:46 pm
by rockedge
good work!

Re: High CPU when camera is disconnected

Posted: Sun Jan 29, 2017 12:40 pm
by DoubleHP
Don't have direct relation between /dev/shm use and cam unreachable. Events may happen at the same time, or not.