OLD ZM box - Questions

[Ruler]

Apologies if this is the wrong section. I honestly didn't know where to post this and I have a hunch I'm going to end up installing the latest version of ZM; rapidly losing hope of repairing the present system.


Background:

I have an old zoneminder box (installed in 2004!) running Slackware 10 and ZoneMinder 1.19.somethin. Both of these were current at the time and nothing has been touched on the box since. However, one of the 4 IDE drives in software RAID0 died and another was experiencing a rapid increase in the number of bad sectors indicative of impending failure. Unfortunately, I was unable to find a source to buy IDE drives large enough to simply do a direct-replacement and so bought a Silicon Image 3114 based SATA controller with 2 large SATA drives. I also had to replace the power supply as there were no SATA connectors on the old one.

Here's where things take a turn for the worse. The 2.4.26 kernel didn't support the SATA controller. I updated to the 2.4.29 kernel (which I have running on another ZM box) and compiled support for the card as a module. I could then see the drives, but problems remain. After changing /etc/raidtab to reflect the new drive layout, I ran mkraid /dev/md0 to create the array. This went fine, as did mkreiserfs /dev/md0 to create a file system on the RAID0 virtual device. I could mount the device, it showed the correct size of both drives combined, and I thought all was going to be well.

Until I rebooted. Everything works fine when I create the RAID device and file system then mount the device and copy files to it. Upon rebooting, the drives are seen by the system as they should be, but all the RAID information seems to disappear. The device is unmountable, the system saying it can't find a file system on it and that there's no superblock. If I try creating a file system on the device after rebooting without re-creating the raid, it tells me that the device is 0 bytes. I've repeated this several times, changing little stuff each time in an attempt to get it functional, but have been unable to solve it. I've since updated the kernel to 2.4.33, the latest in any distribution of slackware, compiling from source and installing the kernel as an option. Either drive works fine if I create a file system on it and reboot; it's just when they're concatenated together into a RAID-0 that problems occur.

At this point, I don't know what to do next. It would be ideal to simply get the RAID working with the SATA drives as everything was working perfectly apart from the drives that were physically failing, but I'm at a loss of what to try next. (This in itself is testimony to the solidity of linux/zoneminder. This system has been basically fully loaded, continually recording 16 cameras for the past 9 years, and the only problems have been power outages long enough to cause the UPS to drain and now hard drives having problems.) I certainly wouldn't mind running a more recent kernel, assuming everything in the machine works with the newer kernels, but don't know if the binaries on the machine will work with a 2.6.x or 3.x kernel without recompiling/reinstalling everything. Plus, it'd still be an old system, so it'd probably be quicker/easier to install a new distro from scratch and install/configure from there. (I really hate to lose all the ZM configs as I spent considerable time to get everything balanced so as not to overload the machine.) I also realize that my knowledge is very dated, not having set up a ZM box since 1.21 and have been using BSD for the past several years, so there will definitely be a considerable learning curve both for the system and software.



Questions:

Does anyone have ideas of what I could try in order to get the RAID functional with both SATA drives with the machine as it is?

What is the quickest/easiest way to install a box with ZoneMinder at the present time? I remember working on the last one I did for a week before it was done; is this true of current revisions?

If I were to set up the latest version of ZM on a box with a current distro, is it possible/reasonable to migrate the configuration from the present box, understanding that ZM cannot be started on this machine due to the RAID not being there anymore? (The database exists, but the events & web interface files were all on the RAID.)

[knight-of-ni]

I admit I read your post very quickly so I may have missed something.
As long as you have your image files and database on a volume you can get access to then all is not lost. So back those files up if you have not already.

Looks like you made the correct choice and created a linux raid volume, rather than try to use the (fake) raid bios in the 3114.

However, have you put the 3114 into IDE mode? You defintely don't want the controller in RAID mode and AHCI mode might not work either for such an old system. If I recall the 3114 bios, you may have to flash it with a special non-raid bios to get it out of raid mode (see the Si website).

Also, how big are those SATA drives? You may run into issues if either drive or the raid volume exceeds 2TB.

[Ruler]

Thanks for the tip! I had not thought of putting the card in IDE mode, so I downloaded the IDE BIOS and flash utility and did so. Unfortunately, what appeared to be the same exact problem cropped up - I could create the RAID device and mount it, but a reboot saw the RAID disappear. However, I noticed that now I was able to start the RAID and mount it without recreating it. Played around in the startup scripts for a while until I found where the fsck happens at boot and added /sbin/raidstart /dev/md0 before it. VOILA! The system sees and mounts the RAID file system at boot. Must be the autodetection fails when using software RAID through this card, even though the partitions are set to be type FD (autodetect). Problem fixed and the server's back up & running as it was!

Or so I thought. After loading the web interface, I noticed that all of the devices are in red indicating that they aren't functional. Instead of reading /dev/video0(1), they all had an extra /dev/video on the front, so each would be /dev/video/dev/video0(1) Didn't touch the ZM configuration so I'm unsure of why this changed; changed them and they remained red, so something else is wrong somewhere. (Was hoping the permissions on the device nodes just got screwed up, but that's a negative - all are set to RW for everyone. Maybe it's due to the fact I'm running a different kernel???) I've yet to troubleshoot this much, but wanted to report back with what I found on the RAID. Hopefully, this new problem will be easier to solve than the previous.

[knight-of-ni]

Glad you made progress. However, there is something that still concerns me. When you were running the silicon image raid bios, the controller may have written (fake)raid meta data to the drives. This can cause the symptoms you described. What happens is the Linux kernel (dmraid driver) sees the meta data and thinks you have a fake raid volume rather than a linux software raid volume.

The way to fix it is to use dd to zero out the raid metadata (if you know where it is at on the disk), or use dd to zero out the entire disk and start over. Last time I did this, I just zeroed out the whole disk, but I'm sure with some googling you could figure out exactly where Silicon Image stores its meta data and zero out just that portion.... it will be at the very beginning or very end of the disk.

Anyway, I know you said you got the system to boot, but I thought I'd mention this potential snafu in case the issue of the missing raid volume comes back.

As far as the zoneminder monitors not coming online, I must have missed something. Did you upgrade the os? Did you try to upgrade zoneminder to a newer version? Generally, what software has changed since the machine last ran fine?

[Ruler]

Got it. The bttv module was renamed between kernel revisions, so when I copied the .config from the old kernel to the new, it didn't know to build it. Set it to build, rebuilt & installed the kernel, rebooted, and everything is working.

Thanks again for the help.

[Ruler]

OK, so I left last night with everything running and sent out a broadcast that the cameras were back up. Got in this morning to have 614 different people leave me messages that there's nothing on the cameras. Reminded me of the opening scene of Office Space with the memo...

Here's where it gets very, very, very odd, odder than it was. I didn't touch anything with ZoneMinder except to change the device nodes (which I changed back after the bttv driver was built/loaded). The events are being recorded fine - the event directories are being created and there are jpg's in each being saved at the proper frame rate. I confirmed that the cameras are feeding the card correctly by examining the image file directly - they show what the camera sees. The problem comes in streaming events - both the 'live' and recorded events don't stream and instead show up as broken link icons in the web interface.

The only thing I can think of at this point is that a dependency of ZM is not functioning because it was compiled/installed under kernel 2.4.26 and I'm now running kernel 2.4.33. I'm going to start going through them one by one, rebuilding and reinstalling each. This is a long shot IMO, but it's all I've really got left unless someone else has any ideas.


Good point about the RAID BIOS in the card maybe leaving something behind on the disks - I'll wipe while I compile.

[knight-of-ni]

What do your log files say?

[Ruler]

Sorry for the delay - we just got done with an off-site sale and I had to handle setup/tear-down of that. I tried updating ZM thinking that would help, but only ended up completely hosing the machine - database errors in ZM logs, capture daemons refused to start, etc. Luckily, I created a bit-wise image of the boot drive (made with dd) before doing anything.

After restoring the machine to that point in time, I built an updated kernel with SATA support, got the RAID functioning as it should, and copied the bare file structure needed by ZM to it. (I copied the entire /video directory to /root before starting ZM for the first time back in 2004, just in case this exact thing happened.) The ZM daemons are running and recording events as they should, but I'm unable to see any streams, whether live or recorded.


There are no errors in the ZM logs from what I can see. The apache error_log has 'Premature end of script headers: /video/cgi-bin/zms' in it. When I change the option in ZM to make it use nph-zms, this error goes away, but there's still nothing but a broken image link in place of the stream. It does not matter if you try a live stream or a recorded event. However, I know it's recording as I can get the stills of previously recorded events. The thumbnails each appear as a broken image when on the stills page, but clicking on one shows the full image as grabbed by the camera. This is the same behavior on several different browsers, each of which was able to stream before the drive replacement. (I know it's got to be something simple, but the solution eludes me.)


*edit - clarified behavior*

[Ruler]

I have a few more clues. (I think.)

If I view the streamed image, I get the following URL as the source of the image:

http://vss/cgi-bin/nph-zms?mode=jpeg&mo ... 0&maxfps=5

(vss is the internal name of the machine on my local network - short for Video Surveillance System.) Interestingly enough, the following is shown in the browser when I do this:

Code: Select all

Server: ZoneMinder Video Server/1.19.5
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Mon, 30 Sep 2013 19:33:19 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache

So there's no jpeg data being sent by ZM; makes sense that a broken image link is shown given that this is the case. If I remove the nph- from the URL, I get an apache server configuration error page, which is when the apache error log has 'premature end of script headers' written to it. This is curious, as the ZM config should not have changed when I replaced the failed drive - the database is stored on the boot drive rather than the RAID. Nothing changed as far as the monitor configurations or setup. Same thing goes for the rest of the machine configs - apache, php, etc are all on the boot drive and should not have been altered by replacing the failed RAID drive. (The RAID has /video on it. All that's under this is html-root and cgi-bin, both exclusively for ZM to use for event recording.)

My guess is that there's something minor that needs to be changed to make /cgi-bin/zms function as it should and that once this is done, the problems will disappear. (I figure it must be something like this because it was working fine before the RAID drive failed using /cgi-bin/zms) Having discovered this, I've got half a memory of running into something like this when first setting up the machine, but unfortunately don't remember the details and didn't put it my notes from the time; it was nearly 10 years ago I last changed anything on this box!

I've been trawling through the forums, hunting for clues, but so far have come up empty. There are quite a few posts about this same type of issue, but none I've found with these exact symptoms. The client machines are running Firefox and have not changed since before the drive failure; cambozola is not installed, nor has it ever been.



Also, could somebody please move this thread to the 'previous versions' forum? My trying to upgrade ZM was enough of a disaster and it's so close to working right now that I do not believe I'll be updating to 1.26, at least not at this time.

[Ruler]

So I had the thought of simply recompiling ZM, thinking it might be the change in kernel version causing the trouble. Turns out that under kernel 2.4.33, the binaries are ~6.1 meg each instead of ~700k each and that both zma and zmf go into defunct status immediately after starting, one pair for each of the 16 cameras on the system.

Of course it's no longer recording, so I'm going to again try updating ZM to the latest version before imaging the drive to what it used to be before the drive failure - doesn't work now anyways, so there's nothing to lose if the update hoses the database again.

[knight-of-ni]

I don't recall if you mentioned what distro you were using, but if you are reloading the drive again, you could consider installing CentOS 6. The advantage being that I've got rpms that will install the latest version which means no compiling.

I'm sure it'll run on your hardware especially if you do a minimal install with no gui.

[Ruler]

I don't recall if you mentioned what distro you were using, but if you are reloading the drive again, you could consider installing CentOS 6. The advantage being that I've got rpms that will install the latest version which means no compiling.

I'm sure it'll run on your hardware especially if you do a minimal install with no gui.

Where would one find these RPMs? (Downloading CentOS 6.4 as I type this. Already printed out each monitor's settings and all the ZM options in preparation of starting over. )

[knight-of-ni]

If you want to go with a minimal install (no gui) then you'll need to download the 4.x GB dvd image, not the live image (something I learned the hard way).

zoneminder rpms are here:
http://www.zoneminder.com/forums/viewto ... =9&t=19119

good luck.