ZoneMinder Periodic Self Shutdown

[BlackAndChrome]

I've got a zoneminder setup on ~10 Foscam cameras that keeps shutting itself down. I can access the web interface and start it again and it works fine for a week or two but if I don't check it daily it obviously doesn't record video until restart. Only other user does not have administrative permission and cannot shutdown the system. I've attached snip of the log for reference, several of the cameras are intermittent so I get fairly constant connection failure messages

Hopefully this is a simple fix, I don't get much time to toy with this system anymore.

Thanks,
-Nick

[iconnor]

I suggest using monit to monitor and restart it.

We have had a few reports of this, with no useful information to help us figure it out. So to start, use monit.

I use the following:
check process zmdc.pl with pidfile /run/zm/zm.pid
if failed unixsocket /run/zm/zmdc.sock then restart
group zm
start program = "/bin/systemctl start zoneminder"
stop program = "/bin/systemctl stop zoneminder"
#if 4 restarts within 20 cycles then timeout

[BlackAndChrome]

Thanks,
I'll have a chance to play with it this weekend.

So is this an ongoing problem in 1.30 or something that may be resolved with an upgrade?

-Nick

[knight-of-ni]

The cases I have seen are caused by overloading the hardware you are running on.
What are the specs of the machine?
What does the average cpu, memory, and disk load look like?
What would be the most help is watching your resources just prior to zoneminder stopping.

[BlackAndChrome]

Xeon e3-1220 v3
8GB DDR3
WD 3TB Red

Been a while since I looked but max loading on the CPU was 1.2 according to ZM ( so ~30% on a 4 core CPU?) and motherboard still has low power CPU throttling on, RAM was <3GB and I have not made a mental note of disk usage but my resolutions for all ~7 cameras that work at any time are 1.3MP or less.

[knight-of-ni]

I can barely read the text in your screenshot, but those "No route to host" messages in your log mean zoneminder no longer has a tcpip connection to the camera. That is a network or camera problem.

What about your ramdisk? How full is that? df -h /dev/shm
What are the resolutions and frame rates of your cameras?
Have you placed any values into the MAX FPS and Alarm MAX FPS fields?

It looks like these cameras are set up with a source type of Remote HTTP. Do these cameras work in a different configuration such as a source type of ffmepg?

Recommend you continue to monitor the system and look for patterns in your logs. For example, does the system stop only after a series of "no route to host" messages? We need to know the series of events leading up to the failure.

[BlackAndChrome]

I'll do some digging this weekend and reply back with more of the specifics requested. Sounds like ZM is in a stop condition again...

Resolutions are all 1.3MP or less, 4-5FPS.
Connections are all remote FFMPEG to my knowledge but I do see the HTTP so I will have to double check and see if maybe one is incorrect.
FPS is set in the cameras, MAX FPS in ZM is set 1 frame higher to cap the framerate when the camera drops out.

This system was supposed to be a test using some cheap Foscam fi9821 and similar PTZs that would get replaced with better cameras later, this year has not been good financially so I'm stuck trying to figure out why the existing cameras don't work, whether it is solely a camera problem or a network problem masked by endless camera problems. Wireless almost never worked and even over ethernet I still get frame loss, smearing, and connection loss that I have to try and preclude out. Cameras are not on a VLAN but it's all gigabit hardware and there is only one computer on the network accessing a 1.3Mb internet connection over the same branch. I'm inclined to blame the cameras though.

Thanks,
-Nick

[BlackAndChrome]

Ramdisk shows as follows after restarting
Size: 3.9G
Used:1.2G

Disk usage is several MB/s intermittent when I switch half the cameras to record according to iotop.

There are 3 cameras running HTTP, all are producing video and appear to be configured correctly, I do not believe they function with ffmpeg.

I've attached the newest log, appears very similar.

I did update my system, including ZM to 1.30 and restarted (it had been 105 days). We'll see how long it lasts this time.

Thanks,
-Nick

[knight-of-ni]

As before, those errors indicate a camera or network issue.

We need to see what was going on just prior to the errors occurring.
Instead of a single page screenshot, try pasting the contents of your log files under /var/log/zoneminder into a gist. You may have to enable that first under Options.

[iconnor]

Also, check mysql. I've sometime seen mysql restart and kill all connections. ZoneMinder will stop as a result.

I really do recommend using monit to make sure zoneminder stays running.

[knight-of-ni]

Some of the features which are part of systemd are relevant here.

Your zoneminder service file, probably has the following line in it:

Code: Select all

Requires=mysqld.service httpd.service

What that means is that ZoneMinder will not start unless mysql and http services are started. Unlike the former sys v init, systemd takes that a step further and will start these services for you automatically if they are not already running. If you try to restart mysql or http service, systemd will automatically restart zoneminder. Nice. This fixes a long standing issue where the end user would restart mysql, but forget to restart zoneminder which leaves zoneminder in a failed state.

Keep in mind, however, this feature of systemd also means that, if mysql were to fail for any reason, systemd would automatically shutdown zoneminder the moment mysql failed. This is why iconnor eluded to the fact that you should monitor your mysql server because a failed mysql would certainly account for zoneminder stopping abruptly .

Another feature of systemd might come in handy as well, but I am hesitant to tell you because even though it might make the problem appear to go away, it won't get us any closer to learning the root cause of the problem.

See the systemd documentation: https://www.freedesktop.org/software/sy ... l#Restart=

If you add the line "Restart=always" to your zoneminder service file (don't forget to tell systemd to reload its daemon config), and possibly your mysql service file, then systemd will attempt to restart these services if they stop for any reason. Note, however, if mysql fails in a way that makes it un-startable, then zoneminder will still fail to restart as well.