Events not being written - ZM shows Disk: 85%

[nrml]

Hello All,

So we've been using ZM on this particular machine for about a month now and since yesterday ZM stopped writing events to the hard disk. The firt time we noticed a problem was when we tried to log in to the zm interface and it would just drop us back to the user/pass prompt. Checked logs and mysql was saying:

Code: Select all

090127 21:02:39 guardian: restarting instance mysqld
090127 21:02:39 starting instance mysqld
090127 21:02:39 [Warning] Changed limits: max_open_files: 1024  max_connections: 500  table_cache: 257
090127 21:02:39  InnoDB: Started; log sequence number 0 43655
^G/usr/sbin/mysqld-max: File './mysql-bin.000067' not found (Errcode: 2)
090127 21:02:39 [ERROR] Failed to open log (file './mysql-bin.000067', errno 2)
090127 21:02:39 [ERROR] Could not open log file
090127 21:02:39 [ERROR] Can't init tc log
090127 21:02:39 [ERROR] Aborting

090127 21:02:39  InnoDB: Starting shutdown...
090127 21:02:41  InnoDB: Shutdown completed; log sequence number 0 43655
090127 21:02:41 [Note] /usr/sbin/mysqld-max: Shutdown complete

We eventually got everything started back up and we are now able to view the cameras but I noticed that nothing is being written to the hard disk:

Code: Select all

[/var/www/html/zm/events/24/09/01]# date ; pwd ; ls -lahd *
Wed Jan 28 01:54:45 PST 2009
/var/www/html/zm/events/24/09/01
drwxr-xr-x 26 apache apache 4.0K Jan  1 23:50 01/
drwxr-xr-x 26 apache apache 4.0K Jan  2 23:50 02/
drwxr-xr-x 26 apache apache 4.0K Jan  3 23:50 03/
drwxr-xr-x 26 apache apache 4.0K Jan  4 23:50 04/
drwxr-xr-x 26 apache apache 4.0K Jan  5 23:50 05/
drwxr-xr-x 26 apache apache 4.0K Jan  6 23:50 06/
drwxr-xr-x 26 apache apache 4.0K Jan  7 23:50 07/
drwxr-xr-x 26 apache apache 4.0K Jan  8 23:50 08/
drwxr-xr-x 26 apache apache 4.0K Jan  9 23:50 09/
drwxr-xr-x 26 apache apache 4.0K Jan 10 23:50 10/
drwxr-xr-x 26 apache apache 4.0K Jan 11 23:50 11/
drwxr-xr-x 26 apache apache 4.0K Jan 12 23:50 12/
drwxr-xr-x 26 apache apache 4.0K Jan 13 23:50 13/
drwxr-xr-x 26 apache apache 4.0K Jan 14 23:50 14/
drwxr-xr-x 26 apache apache 4.0K Jan 15 23:50 15/
drwxr-xr-x 26 apache apache 4.0K Jan 16 23:50 16/
drwxr-xr-x 26 apache apache 4.0K Jan 17 23:50 17/
drwxr-xr-x 26 apache apache 4.0K Jan 18 23:50 18/
drwxr-xr-x 26 apache apache 4.0K Jan 19 23:50 19/
drwxr-xr-x 26 apache apache 4.0K Jan 20 23:50 20/
drwxr-xr-x 26 apache apache 4.0K Jan 21 23:50 21/
drwxr-xr-x 26 apache apache 4.0K Jan 22 23:50 22/
drwxr-xr-x 26 apache apache 4.0K Jan 23 23:50 23/
drwxr-xr-x 26 apache apache 4.0K Jan 24 23:50 24/
drwxr-xr-x 26 apache apache 4.0K Jan 25 23:50 25/
drwxr-xr-x 26 apache apache 4.0K Jan 26 23:50 26/
drwxr-xr-x  5 apache apache 4.0K Jan 27 02:10 27/
[/var/www/html/zm/events/24/09/01]#

So long story short I enabled debug and saw this: (one of many)

Code: Select all

Jan 28 01:02:25 camera02 zma_m27[10039]: FAT [Can't mkdir events/27/09/01/28: No space left on device]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Got signal (Aborted), crashing]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Signal address is (nil), from 0xbfffe410 ]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: [0xbfffe410]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: [0xbfffe410]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /lib/i686/libc.so.6(gsignal+0x50) [0x4d435fc0]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /lib/i686/libc.so.6(abort+0x101) [0x4d437851]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /usr/lib/zm/bin/zma [0x8071b68]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /usr/lib/zm/bin/zma [0x8058089]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /usr/lib/zm/bin/zma [0x806c08b]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /usr/lib/zm/bin/zma(__gxx_personality_v0+0x3cc) [0x804afc8]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /lib/i686/libc.so.6(__libc_start_main+0xdc) [0x4d42375c]]
Jan 28 01:02:25 camera02 zma_m27[10039]: ERR [Backtrace: /usr/lib/zm/bin/zma(__gxx_personality_v0+0xa5) [0x804aca1]]
Jan 28 01:02:25 camera02 zma_m27[10039]: INF [Backtrace complete]

With that in mind, I can't understand what's going on, we have a filter for the -30 days which should clear stuff out but it hasn't been 30 days yet. Another thing that puzzles me is the fact that the interface says:

Code: Select all

Load: 0.29 / Disk: 85%

Code: Select all

[/var/www/html/zm/events/24/08/12]#  df -h /var/www/html/zm/events
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             1.4T  1.1T  201G  85% /var/www/html/zm/events
[/var/www/html/zm/events/24/08/12]# 

so why does it complain about space?

thanks for your help, I can provide any more logs if needed.

Thanks again

/gabe

[cordel]

What happens if you drop the database and create MyISAM in it's place?
This is a hunch:
I think you might have deadlocked a row which might have fubared the database.
or
What space do you have for your database? Innodb is a hog on the drive.
df -h /var/lib/mysql

[Paranoid]

There are 2 different limiting factors when storing files, ones the disk space and the other is the inodes.

Try running:

df -i

This will tell you how many inodes you have used and how many you have left.