ZoneMinder Forums

Posted: **Thu Mar 21, 2019 3:11 pm**

Hello,
I've zoneminder 1.30.4+dfsg-2~bpo9+1 on Debian 9, installed from backports. It works with php-fpm7.0 on Nginx.
I've installed zoneminder about 6 weeks ago, configured, installed the webcam, everything was fine for what I had tested.
Since maybe 10 days ago zoneminder keeps crashing.
Here something from the log during the crash:

Code: Select all

Mar 21 15:56:51 mycomputer zmaudit[16269]: FAT [Can't execute: Lost connection to MySQL server during query]
Mar 21 15:56:51 mycomputer zmaudit[16269]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Mar 21 15:56:51 mycomputer zmdc[16218]: ERR ['zmaudit.pl -c' exited abnormally, exit status 255]
Mar 21 15:56:51 mycomputer zmdc[16218]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Mar 21 15:56:53 mycomputer zmwatch[16274]: FAT [Can't execute: MySQL server has gone away]
Mar 21 15:56:53 mycomputer zmwatch[16274]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
Mar 21 15:57:27 mycomputer zmc_dvideo1[16262]: INF [gspca_sn9c20x: 21000 - Capturing at 19.61 fps]
Mar 21 15:57:47 mycomputer zmfilter[16263]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
Mar 21 15:58:18 mycomputer zmc_dvideo1[16262]: INF [gspca_sn9c20x: 22000 - Capturing at 19.61 fps]
Mar 21 15:58:47 mycomputer zmfilter[16263]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
Mar 21 15:59:09 mycomputer zmc_dvideo1[16262]: INF [gspca_sn9c20x: 23000 - Capturing at 19.61 fps]
Mar 21 15:59:47 mycomputer zmfilter[16263]: FAT [Can't execute 'SELECT * FROM Filters WHERE Background = 1 and(  AutoArchive = 1#012                or AutoVideo = 1#012                or AutoUpload = 1#012                or AutoEmail = 1#012                or AutoMessage = 1#012                or AutoExecute = 1#012                or AutoDelete = 1#012             ) ORDER BY Name': MySQL server has gone away]

I don't have a clue about what's exactly going on, but I can add that I suspect the crash first occurred after a recent upgrade of the whole php suite, included php-fpm7.0...
Losing the connection to the db is weird, maybe there's a timeout to increase somewhere, I don't know

Thank you for the help

Posted: **Thu Mar 21, 2019 9:49 pm**

perhaps you are running out of mysql connections. It defaults to 100 I think... each monitor can use 3 or 4 connections, plus more for every viewer.

I improved this is 1.32, including showing how many are in use in the ui. Check your mysql logs.

Posted: **Fri Mar 22, 2019 9:32 am**

Thank you for your answer.
In my my.cnf I didn't set the parameter maxconnection. At the moment I only have one camera in function, to make it easier to investigate the problem. The mysql server log, which I've just enabled, shows:

Code: Select all

3 Query     insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( '1553246596.80221', 'zmfilter', '8658', '-2', 'ERR', 'Can\'t execute filter \'SELECT E.*,\n       unix_timestamp(E.StartTime) as Time,\n       M.Name as MonitorName,\n       M.DefaultRate,\n       M.DefaultScale\n         FROM Events as E\n         INNER JOIN Monitors as M on M.Id = E.MonitorId\n          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100\', ignoring: MySQL server has gone away', 'zmfilter.pl', NULL )

On MariaDB there's only the ZM DB it seems weird to me that it's not able to bear one db. And moreover, as I've before pointed out, when I installed it, it was working...

Thanks

Posted: **Sat Mar 23, 2019 1:16 am**

Run mysqltuner to see recommendations. Note that the value may not be in the my.cnf or one of the other config files for the database server but the server is using a default value (151 for Mariadb). It is possible the my.cnf is pretty bare. You can add a value to the file:

[mysqld]
max_connections=[desired new maximum number]

See: https://www.ionos.com/community/hosting ... ons-error/

Posted: **Sat Mar 23, 2019 4:13 pm**

Thank you for the tip about mysqltuner, I've changed some values according to the suggestions, and also set max_connections=200 but the server keeps crashing

Code: Select all

zmaudit[15538]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
zmdc[15495]: ERR ['zmaudit.pl -c' exited abnormally, exit status 255]
 zmdc[15495]: FAT [Can't execute log entry 'insert into Logs ( TimeKey, Component, Pid, Level, Code, Message, File, Line ) values ( ?, ?, ?, ?, ?, ?, ?, NULL )': MySQL server has gone away]
 zmfilter[15530]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
 zmwatch[15545]: FAT [Can't execute: MySQL server has gone away]
 zmfilter[15530]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
 zmfilter[15530]: ERR [Can't execute filter 'SELECT E.*,#012       unix_timestamp(E.StartTime) as Time,#012       M.Name as MonitorName,#012       M.DefaultRate,#012       M.DefaultScale#012         FROM Events as E#012         INNER JOIN Monitors as M on M.Id = E.MonitorId#012          where not isnull(E.EndTime) and ( E.Archived = 0 and 66 >= 95 ) order by E.Id asc limit 0,100', ignoring: MySQL server has gone away]
 zmfilter[15530]: FAT [Can't execute 'SELECT * FROM Filters WHERE Background = 1 and(  AutoArchive = 1#012                or AutoVideo = 1#012                or AutoUpload = 1#012                or AutoEmail = 1#012                or AutoMessage = 1#012                or AutoExecute = 1#012                or AutoDelete = 1#012             ) ORDER BY Name': MySQL server has gone away]
 zmc_dvideo1[15528]: INF [gspca_sn9c20x: 5000 - Capturing at 2.72 fps]
 zmc_dvideo1[15528]: INF [gspca_sn9c20x: 6000 - Capturing at 2.60 fps]

mah...

Posted: **Sun Mar 24, 2019 12:44 pm**

200 may not be enough. I have a server with a lot of folks viewing montage so I set connections to 1000.

Check your memory especially tmpfs. Linux uses half of the RAM for tmpfs and Zoneminder can fill that space, /dev/shm, if you use high resolution camera settings.

Posted: **Sun Mar 24, 2019 3:31 pm**

Thank you for your answer. Unfortunately it doesn't matter how high I set the value of max_connections, it keeps crashing. I also checked the temporary fs and and it's still almost empty. Please note that I'm currently running only a small usb cam at low resolution (640px), I would wonder if such environment could cause all those troubles to my system (but I'm not a zm expert).
What I don't understand (well, among the other issues) is why before the same system was working without problems. The only change I'm aware of, is the update of php, which let me think this could be the cause, although I don't understand how.
Thanks.

Posted: **Fri Mar 29, 2019 4:02 pm**

I've stopped logging to the web gui and the db; now it works without crashing, and when it does, it restarts without doing anything

Posted: **Sat Mar 30, 2019 5:38 am**

ebenezer191 wrote: ↑Fri Mar 29, 2019 4:02 pm I've stopped logging to the web gui and the db; now it works without crashing, and when it does, it restarts without doing anything

Your comment reminded me that I had to do the same sort of thing. My initial ZM 1.30.4 (on Ubuntu 18.04) setup ran fairly well at first with just one cam with logging to DB enabled. But, after adding several more cams ZM became increasingly unstable with odd monitor errors and race conditions. I eventually suspected some sort of edge condition related to DB contention/conflict. Tried additional DB tunings (via mysqltuner) multiple times over several weeks but to no avail. My system has a 6 core 12 thread CPU, 16 gigs of RAM and ample fast disk storage - the ZM stability issues were not related to lack of resources or bandwidth. Anyway...

To debug further, I planned to do the typical exercise: systematically disable/enable some things/settings and observe results. Turning off all logging to DB was the first thing I tried and THAT immediately restored stability.

Also (after the above), based on a bug report I read someplace (can't find a reference now) I also I set both LOG_LEVEL_SYSLOG and LOG_LEVEL_FILE to Info. This is actually redundant, but doing this (supposedly) avoids tripping an odd bug that can result in a weird ZM exception error. Doing this also meant I had to make some changes related to how log files are managed...

I added some rsyslog rules to assure all rsyslog related ZM log entries actually go to /var/log/zm.log. To do this I added a /etc/rsyslog.d/10-zm_log.conf file, with contents like this:

:programname, startswith, "zm" /var/log/zm.log
& stop
:programname, isequal, "check" /var/log/zm.log
& stop
:programname, isequal, "check2" /var/log/zm.log
& stop
:programname, startswith, "web_" /var/log/zm.log
& stop

This ZM logging setup does result in redundant logging to multiple logs in /var/log/zm AND a "unified" log /var/log/zm.log. Not exactly elegant, but my ZM setup remains 100% stable with this.

For log rotation, my /etc/logrotate.d/zonenminder config file contents:

#
# Zoneminder logs directly to files in /var/log/zm/
# (not via syslog/rsyslog). Thus rotate must be done
# via Zoneminder native log rotation facility.
#
/var/log/zm/*.log {
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
/usr/bin/zmpkg.pl logrot >>/dev/null 2>&1 || :
endscript
}
#
# Zoneminder uses syslog/rsyslog to log to /var/log/zm.log.
# Thus rotate is handled the standard way for syslog/rsyslog logs.
#
/var/log/zm.log {
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

My ZM 1.30.4 setup has been rock stable (over a year now) ever since making these changes.

Note: the examples above are for an Ubuntu 18.04 system.

Posted: **Sat Mar 30, 2019 2:26 pm**

Thank you

ZoneMinder Forums

Zoneminder keeps crashing

Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing

Re: Zoneminder keeps crashing