diagnosing missing event files. or not saving files?

[eddyhkim]

I've got a machine running zoneminder 1.22.2 on ubuntu dapper.

It's got 3 cameras connected to a 4 chip bt card. It had been working without problems for about 5 (?) months, but a month back, I noticed some really old events with wierd timestamps, and so I deleted them. This caused all events to stop recording, and so basically I deleted all events off the filesystem, and re built the mysql database using the original db script as found in db/zm_create.sql

I re-added all the cameras, and have them on modetect. and it appeared things were back to normal. streaming and event viewing worked, as far as I could tell.

Now:

streaming works on all three cameras.

The event viewing feature works on monitor 1 only. it doesn't work on the two other cameras i'm concerned with.

kiosk-video:/var/www/events$ ls -l
total 1188
drwxrwxrwx 11248 www-data www-data 180224 2006-10-09 18:03 1
drwxrwxrwx 32000 www-data www-data 499712 2006-10-08 11:19 2
drwxrwxrwx 32000 www-data www-data 491520 2006-10-02 17:43 3
drwxrwxrwx 2 www-data www-data 4096 2006-09-25 16:01 4
drwxrwxrwx 2 www-data www-data 4096 2006-09-25 16:02 5
drwxrwxrwx 1523 www-data www-data 24576 2006-10-02 13:44 6
lrwxrwxrwx 1 www-data www-data 1 2006-09-25 15:56 back -> 3
lrwxrwxrwx 1 www-data www-data 1 2006-06-19 01:24 front -> 1
lrwxrwxrwx 1 www-data www-data 1 2006-09-25 16:03 pos -> 6
lrwxrwxrwx 1 www-data www-data 1 2006-10-09 17:01 register -> 2
lrwxrwxrwx 1 www-data www-data 1 2006-09-25 16:01 storage1 -> 4
lrwxrwxrwx 1 www-data www-data 1 2006-09-25 16:02 storage2 -> 5

the first three cameras are monitors 1, 2 and 3.

There are plenty of event subdirs in 2 and 3, but I don't know if they are current up to today. From the event viewer, I did notice the last viewable event was from 2006-10-08 (yesterday mid-morning) but there was nothing I did to the system at the time.

In the event viewer, if I click on an event, the viewing window shows a broken link. e.g. I try to click on event 108730 for monitor 2. It doesn't work because that event doesn't exist in the file system. I also checked all the other subdirs, and those don't have it either (ls */108730 returns no matches).

I don't see anything in the syslog that would clue me in. No WAR, just INF lines. I have 299Gigs free. system load is generally low, except when msyqld gets hit by some processing.

Any ideas on where to start looking for the cause of this problem? how do i reset the daemons/db to a good state without wiping out everything?

To reiterate, events do work for monitor 1 only. real-time streaming works for all monitors.

TIA,
Ed

[eddyhkim]

also, this machine is on a nightly reboot via cronjob.

I stopped zm and did a mysqlcheck -A to repair all the databases.

nothing reported wrong, and the events are still unviewable/not there.

[jameswilson]

so your console shows 0 events?

[eddyhkim]

nope, the console shows all the events. I have a cameras pointed at the cash register, front and back. All of them show events, and alarms seem to be triggered as before.

It's when i click on the event to view it, that the event viewing window has a broken image. And when I try to track down the event number in the file system it's not there. a ps listing shows multiple zma and zmc processes. There isn't anything in the syslog showing them dying or anything wierd that I can determine.

Right now, i'm deleting the register monitor, and as soon as it's done, i'll re-add it. It is critical to be able to review this particular camera; I can't wait on this one. I already saw something fishy realtime, but was unable to go back to review the whole event.

The back monitor however I'm leaving in hopes I can get it fixed. What could be causing the capture daemon to not be able to save the file? Keep in mind I have plenty of disk space on that partition and it had been working fine until a couple of days ago. Is there some way to reset the process without deleting and re-adding?

----------

I totally forgot that under /tmp are a bunch of log files. So I checked them, but the only thing that seems related is in zmaudit.log

a bunch of lines like:
10/09/06 23:17:29.125758 zmaudit[5169].INF [deleting]
10/09/06 23:17:29.139538 zmaudit[5169].INF [Database event '2/110054' does not e
xist in filesystem]
10/09/06 23:17:29.140015 zmaudit[5169].INF [deleting]
10/09/06 23:17:29.147105 zmaudit[5169].INF [Database event '2/110060' does not e
xist in filesystem]
10/09/06 23:17:29.147548 zmaudit[5169].INF [deleting]
10/09/06 23:17:29.158762 zmaudit[5169].INF [Database event '2/110051' does not e
xist in filesystem]
10/09/06 23:17:29.160617 zmaudit[5169].INF [deleting]
10/09/06 23:17:29.168347 zmaudit[5169].INF [Database event '2/110057' does not e
xist in filesystem]
10/09/06 23:17:29.168798 zmaudit[5169].INF [deleting]
10/09/06 23:33:03.218454 zmaudit[5169].DBG [Found database monitor '1']
10/09/06 23:33:03.686498 zmaudit[5169].DBG [Got 11484 events]
10/09/06 23:33:03.686719 zmaudit[5169].DBG [Found database monitor '2']
10/09/06 23:33:05.016227 zmaudit[5169].DBG [Got 32006 events]
10/09/06 23:33:05.016470 zmaudit[5169].DBG [Found database monitor '3']
10/09/06 23:33:06.316727 zmaudit[5169].DBG [Got 31998 events]


but it looks to be routine, not a sign of a problem, since it appears for all the monitors that have recording.. I only see .INF or .DBG lines. no WAR or ERR in there.

[zoneminder]

This is really odd. Can you confirm that events are generated in the DB but that the directory for the event images is not?

It may help to stop zmaudit just to make sure that it is confusing the issue. Do

Code: Select all

zmdc.pl stop zmaudit.pl -c

to stop it.

[jameswilson]

there is an issue with mocord/record that affects stills, but the stream plays back ok, and the images are still there. Just checking you dont just have an issue with still playback

[eddyhkim]

phil,

I stopped the audit process as you suggested, but don't know what i should be looking for as a result of this command.

monitor 1 (named front) is going into/back/left alarm status from the /var/log/messages file. grepping for just the front related messages, there is just normal looking stuff. i can currently view saved events using mozilla from nov 6 but not 7, 8 or 9th, so it appears that something happened on either the 6th or 7th to cause a problem.

Nov 9 10:19:38 kiosk-video zma_m1[5639]: INF [front: 2872 - Left alarm state (198165) - 57(37) images]
Nov 9 10:19:47 kiosk-video zma_m1[5639]: INF [front: 2891 - Gone into alarm state]
Nov 9 10:19:55 kiosk-video zma_m1[5639]: INF [front: 2906 - Gone into alert state]
Nov 9 10:19:58 kiosk-video zma_m1[5639]: INF [front: 2913 - Gone back into alarm state]
Nov 9 10:20:21 kiosk-video zma_m1[5639]: INF [front: 2958 - Gone into alert state]
Nov 9 10:20:22 kiosk-video zma_m1[5639]: INF [front: 2961 - Gone back into alarm state]
Nov 9 10:20:41 kiosk-video zma_m1[5639]: INF [front: 2999 - Gone into alert state]
Nov 9 10:20:42 kiosk-video zma_m1[5639]: INF [front: 3000 - Processing at 1.71fps]
Nov 9 10:20:46 kiosk-video zma_m1[5639]: INF [front: 3009 - Left alarm state (198169) - 128(98) images]
Nov 9 10:20:54 kiosk-video zma_m1[5639]: INF [front: 3023 - Gone into alarm state]
Nov 9 10:21:19 kiosk-video zma_m1[5639]: INF [front: 3074 - Gone into alert state]
Nov 9 10:21:21 kiosk-video zma_m1[5639]: INF [front: 3078 - Gone back into alarm state]

This is a regular occurance on two different systems all running the same versions of ubuntu/zm/apache/mysql

if i delete the monitor and add a new one, it fixes the problem (temporarily for a few weeks) but it comes back. What else should I do to help diagnose this?

Right now it is nov, 9th, 10:35 am.

viewing the realtime view of "front" works.

If I try to view a recently recorded event

javascript: eventWindow( '/index.php?view=event&eid=198197&trms=3&attr1=MonitorId
&op1=%3D&val1=1&cnj2=and&attr2=Archived&op2=%3D&val2=0
&cnj3=and&attr3=DateTime&op3=%3E%3D&val3=-1+hour
&sort_field=DateTime&sort_asc=1&page=1', 'zmEvent', 416, 408 );

Here's a cut and paste from the event window for this event.

198197 Event-198197 front Motion 11/09 10:25:50 5.56 32 12 75 6 8

this means the event is in the db, i assume. there are no errors or warnings re this monitor id or event in syslog. I did a mysqlcheck and no corruption was found.

if I look on the file system for the file:
kiosk-video:/var/www/events/front$ file 198197
198197: ERROR: cannot open `198197' (No such file or directory)

the directory for the event doesn't even exist.

a tips on debugging this? i'm perfectly comfortable editing code if you want to put additional stuff for debugging. Unfortunately I don't have much time to devote to figuring out the codebase myself at the moment.

Thanks,
Ed

[Ellmegamurks]

Hi, Ed,

i hope it's not too late;

i think, it might be possible that you ran into the "32000 entries per directory" - limitation of the ext3-FS, as your logs mentioned something like that.

I advanced to ReiserFS on an 250G-Drive (apparently i had to - my drive went dead) and kept from using Ext3 on my disk as an FS for storing events for zm...

I hope this might be helpfull...

(changing FS as you have to change {still alive, but damaged} harddisks is just fun)

Eric

[ddufresne]

I started losing capture footage at the 32000 Events benchmark. I'm paring down the saved events and this has restored the captures.

I'm paring down the 'Purge When Full' filter to keep the size well below 32000 Events.

Thanks for the posting!