zm crashed due to automatic sys updates

[weave]

Woke up this morning and wondering why my Zoneminder server was offline.

Dug around in the logs and determined it was because MySQL was down, and it was down due to automatic updates were being applied and that triggered mysql to get restarted. It was down long enough to kill Zoneminder....

This is on rev 1.30.4 and Ubuntu 16.04

So wondering if somehow Zoneminder can be changed to wait and retry after a database error to deal with an issue like this.

Failing that, any ideas how I might be able to mitigate this risk somehow? Or should I just kill automatic updates, do them manually, and check my services afterwards?

Below are my log records, if that's helpful at all...

Code: Select all

Oct 24 06:04:20 luray systemd[1]: Stopped MySQL Community Server.
Oct 24 06:04:22 luray systemd[1]: Reloading.
Oct 24 06:04:22 luray systemd[1]: Started ACPI event daemon.
Oct 24 06:04:22 luray systemd[1]: Started CUPS Scheduler.
Oct 24 06:04:25 luray zmwatch[31171]: FAT [Can't execute: MySQL server has gone away]
Oct 24 06:04:25 luray zmwatch[31171]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Oct 24 06:04:25 luray zmdc[31085]: ERR ['zmwatch.pl' exited abnormally, exit status 255]
Oct 24 06:04:25 luray zmdc[31085]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Oct 24 06:04:25 luray systemd[1]: zoneminder.service: Main process exited, code=exited, status=255/n/a
Oct 24 06:04:25 luray zmpkg.pl[21955]: DBI connect('database=zm;host=localhost','zmuser',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/share/perl5/ZoneMinder/Config.pm line 106.
Oct 24 06:04:25 luray zmpkg.pl[21955]: Can't connect to db at /usr/share/perl5/ZoneMinder/Config.pm line 129.
Oct 24 06:04:25 luray zmpkg.pl[21955]: BEGIN failed--compilation aborted at /usr/share/perl5/ZoneMinder/Config.pm line 129.
Oct 24 06:04:25 luray zmpkg.pl[21955]: Compilation failed in require at /usr/share/perl5/ZoneMinder.pm line 33.
Oct 24 06:04:25 luray zmpkg.pl[21955]: BEGIN failed--compilation aborted at /usr/share/perl5/ZoneMinder.pm line 33.
Oct 24 06:04:25 luray zmpkg.pl[21955]: Compilation failed in require at /usr/bin/zmpkg.pl line 48.
Oct 24 06:04:25 luray zmpkg.pl[21955]: BEGIN failed--compilation aborted at /usr/bin/zmpkg.pl line 48.
Oct 24 06:04:25 luray systemd[1]: zoneminder.service: Control process exited, code=exited status=255
Oct 24 06:04:25 luray zmc_m2[31130]: ERR [Can't insert log entry: sql(insert into Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) values ( 1508839465.384089, 'zmc_m2', 0, 31130, 0, 'INF', 'Got signal 15 (Terminated), exiting', 'zm_signal.cpp', 40 )) error(Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))]
Oct 24 06:04:25 luray zmc_m2[31130]: INF [Got signal 15 (Terminated), exiting]
Oct 24 06:04:25 luray zma_m3[31221]: ERR [Can't insert log entry: sql(insert into Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) values ( 1508839465.387224, 'zma_m3', 0, 31221, 0, 'INF', 'Got signal 15 (Terminated), exiting', 'zm_signal.cpp', 40 )) error(Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))]
Oct 24 06:04:25 luray zma_m3[31221]: INF [Got signal 15 (Terminated), exiting]
Oct 24 06:04:25 luray zma_m2[31138]: ERR [Can't insert log entry: sql(insert into Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) values ( 1508839465.391977, 'zma_m2', 0, 31138, 0, 'INF', 'Got signal 15 (Terminated), exiting', 'zm_signal.cpp', 40 )) error(Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))]
Oct 24 06:04:25 luray zma_m2[31138]: INF [Got signal 15 (Terminated), exiting]
Oct 24 06:04:25 luray zmc_m1[31118]: ERR [Can't insert log entry: sql(insert into Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) values ( 1508839465.394486, 'zmc_m1', 0, 31118, 0, 'INF', 'Got signal 15 (Terminated), exiting', 'zm_signal.cpp', 40 )) error(Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))]
Oct 24 06:04:25 luray zmc_m1[31118]: INF [Got signal 15 (Terminated), exiting]
Oct 24 06:04:25 luray zmc_m3[31144]: ERR [Can't insert log entry: sql(insert into Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) values ( 1508839465.397974, 'zmc_m3', 0, 31144, 0, 'INF', 'Got signal 15 (Terminated), exiting', 'zm_signal.cpp', 40 )) error(Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))]
Oct 24 06:04:25 luray zmc_m3[31144]: INF [Got signal 15 (Terminated), exiting]
Oct 24 06:04:25 luray zma_m1[31124]: ERR [Can't insert log entry: sql(insert into Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) values ( 1508839465.406201, 'zma_m1', 0, 31124, 0, 'INF', 'Got signal 15 (Terminated), exiting', 'zm_signal.cpp', 40 )) error(Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2))]
Oct 24 06:04:25 luray zma_m1[31124]: INF [Got signal 15 (Terminated), exiting]
Oct 24 06:04:25 luray systemd[1]: zoneminder.service: Unit entered failed state.
Oct 24 06:04:25 luray systemd[1]: zoneminder.service: Failed with result 'exit-code'.
Oct 24 06:04:42 luray systemd[1]: Starting MySQL Community Server...

[bbunge]

With auto updates you can set the server to reboot at a certain time of the day. Of course you will have to guess when the auto updates will be done...
Might be better to control the updates yourself so you can stop Zoneminder before you update the database server. While updates are usually good to get there have been times when one fried a system. Consider the Windows Fall Creators update.... You really do not want my tirade on Windows...

bb

[weave]

That's an idea. I could cron Zoneminder to shutdown during the day for a few minutes when I'm not as concerned about detection and schedule the reboot during that window. I hear you on the dangers of automatic updates but I suffer from the worse danger of not doing updates because I get distracted by other shiny things.

Thanks.

[iconnor]

You can tell systemd to restart it always. You can use monit to restart it. There are several options.

[opoplawski]

This is pretty crazy that zoneminder exits due to this. All long running software should expect DB connections to go away at times and retry.

[jsylvia007]

I use this script as a cron job that runs every 5 minutes...

Code: Select all

#!/bin/bash

#This script will send an email to root user if the Zabbix Server isn't running.
#This script requires the mutt email program

EMAIL="EMail@Domain.TLD"
SUBJECT="["`hostname`"] - ZoneMinder Server is DOWN!"
BODY="This is an automated message to notify you that the ZoneMinder SERVER on "`hostname`" is DOWN. \n\n\n Attempting Restart... \n\n\n Date and Time: "`date`

#Determine if we're in testing mode...
if [ -e zoneminder_test ]; then
        #echo 'zm is in testing mode'
        exit 0
fi

if [ $( systemctl status zoneminder.service | grep "Active:" | grep "(running)" | wc -l ) -ne 1 ]; then
        #echo "ZM is DEAD!"
	echo -e "${BODY}" | mutt -s "${SUBJECT}" ${EMAIL}
        systemctl restart zoneminder
        sleep 30
        exit 0
fi