Buffer Overrun errors

[elmer]

Not sure if this is an anomaly or signs of a greater issue, so I figured I would pose it to the forums.

I'm running 1.26.5 on Ubuntu Server 12.04. I had been running with two striped 500Gb drives but earlier this week I replaced those with 3 4Tb drives in a RAID 5 configuration using mdadm.

System had been running for over a day and a half recording 26 cameras. I'm not doing any motion detection, every cam is set to Record. Came in this morning, Zoneminder was not running, stopped shortly after 6:30AM. Looking through the logs I saw that every monitor had a buffer overrun error at that same time along with this message:

Code: Select all

2014-01-22 06:35:36.076400	zmfilter	2953	FAT	Can't execute 'select * from Filters where Background = 1 and (AutoArchive = 1 or AutoVideo = 1 or AutoUpload = 1 or AutoEmail = 1 or AutoMessage = 1 or AutoExecute = 1 or AutoDelete = 1) order by Name': MySQL server has gone away	zmfilter.pl

I started zoneminder again from the web interface and it appears to be running without issue. I checked dmesg and there wasn't anything of interest in there.
my /etc/sysctl.d/60-kernel-shm.conf file has the following line:
kernel.shmmax = 536870912

output of free -m:

Code: Select all

 free -m
                 total       used       free     shared    buffers     cached
Mem:          3890       3720        170          0       1237       1744
-/+ buffers/cache:        737       3152
Swap:         4027          0       4027

Any help is greatly appreciated.

[TheKorn]

A WAG would be that something tied up your array around 6:30AM, causing the buffer overruns and mysql to stall catastrophically.

Anything cron'ing off around that time?

Anything interesting in /proc/stat/mdadm ? (I think it's /proc/stat/mdadm, it's been a while since i was in mdadm-land. Basically, what's the array's status?)

Do a "dmesg | grep -i oom". (If nothing comes back, GREAT!)

Finally, what filesystem are you using, do you have any background filters set to run, and do you have deep filesystem enabled?

[elmer]

I don't have anything set to run at that time of day. The path you're looking for in this case is /proc/mdstat. If I cat that, there's nothing interesting, just an active array.

I ran your dmesg command and got nothing back.

I have deep filesystem enabled. The only filter I have enabled is PurgeWhenFull and that is set to trigger at 90%. At this point, I've been running my 4Tb array for less than 48 hours so its <5% full. The filesystem for the array is EXT4. I did not do any filesystem tuning with the array, just let mkfs go with the defaults.

In the meantime I set up monit to watch the zoneminder process and e-mail me if it stops, and restart it. Maybe I should do the same with mysql and apache just in case.