CentOS 5.4 & PV149 & ZoneMinder 1.24.= input dies ov

sck_nogas · Post by **sck_nogas** » Wed Dec 23, 2009 12:43 am

On my CentOS 5.4 server using video4linux2 with a PV149 capture board installed with 8 local wired cameras running ZoneMinder - v1.24.2, I have the following problem...

* Upon reboot, all cameras are good, images appearing, life is happy.
* Over time, some cameras stop being viewable and the only errors in /var/log/messages that indicate that the hardware failed is...
Dec 17 19:40:58 house zmdc[2954]: INF [Starting pending process, zmc -d /dev/video0]
Dec 17 19:40:58 house zmdc[2954]: INF ['zmc -d /dev/video0' starting at 09/12/17 19:40:58, pid = 30269]
Dec 17 19:40:58 house zmdc[30269]: INF ['zmc -d /dev/video0' started at 09/12/17 19:40:58]
Dec 17 19:40:58 house zmc_dvideo0[30269]: INF [Debug Level = 0, Debug Log = <none>]
Dec 17 19:40:58 house zmc_dvideo0[30269]: INF [Starting Capture]
Dec 17 19:40:58 house kernel: bttv0: timeout: drop=0 irq=41702860/41702860, risc=36cc903c, bits: FMTCHG VSYNC HSYNC RISCI
Dec 17 19:40:58 house zmc_dvideo0[30269]: WAR [Capture failure, possible signal loss?: Input/output error]
Dec 17 19:40:58 house zmc_dvideo0[30269]: ERR [Failed to capture image from monitor 1 (0/2)]
Dec 17 19:40:58 house kernel: bttv0: reset, reinitialize
Dec 17 19:40:58 house kernel: bttv0: PLL can sleep, using XTAL (28636363).
Dec 17 19:40:58 house zmdc[2954]: ERR ['zmc -d /dev/video0' exited abnormally, exit status 255]
* So, the image in ZoneMinder is blank (blue screen) and the device in ZoneMinder goes red.
* But not all cameras, the video from /dev/video1 and /dev/video3 works, but not /dev/video0 and /dev/video2?
* Stopping and starting ZoneMinder doesn't solve the issue, but a reboot does.

Interesting factoids?

1) Under Fedora Core [with same PV149] but different motherboard, I never had this problem.
2) It's always the same video devices, but they are on the same board and using the same video chips as the devices that stay working?
3) It is always fixed by a reboot.

So, what I'd like to do is be able to fix the problem permanently, or if not possible detect the problem and fix without a reboot.

Scott
PS> I'm lying... I have 7 cameras working, since I used a VGA cable instead of all 16 pins through so one camera doesn't display. But, I know why.

sck_nogas · Post by **sck_nogas** » Wed Dec 23, 2009 12:56 am

Just digging some more, I decided to look into the /proc filesystem and I notcied that I could trigger some messages for the /dev/video devices that weren't working in the /var/log/messages file.

[sck@house ~]$ ls -ld /proc/self/root/dev/video*
lrwxrwxrwx 1 root root 6 Dec 12 12:28 /proc/self/root/dev/video -> video0
crw----rw- 1 root root 81, 0 Dec 12 12:28 /proc/self/root/dev/video0
crw----rw- 1 root root 81, 1 Dec 12 12:27 /proc/self/root/dev/video1
crw----rw- 1 root root 81, 2 Dec 12 12:27 /proc/self/root/dev/video2
crw----rw- 1 root root 81, 3 Dec 12 12:27 /proc/self/root/dev/video3
[sck@house ~]$ cat /proc/self/root/dev/video? ; sudo tail -100 /var/log/messages| grep bttv
cat: /proc/self/root/dev/video0: Input/output error
cat: /proc/self/root/dev/video1: Device or resource busy
cat: /proc/self/root/dev/video2: Input/output error
cat: /proc/self/root/dev/video3: Device or resource busy
Dec 22 16:52:21 house kernel: bttv0: reset, reinitialize
Dec 22 16:52:21 house kernel: bttv0: PLL can sleep, using XTAL (28636363).
Dec 22 16:52:22 house kernel: bttv0: timeout: drop=0 irq=41702860/41702860, risc=36cc901c, bits: FMTCHG VSYNC HSYNC RISCI
Dec 22 16:52:22 house kernel: bttv2: reset, reinitialize
Dec 22 16:52:22 house kernel: bttv2: PLL can sleep, using XTAL (28636363).
Dec 22 16:52:22 house kernel: bttv2: timeout: drop=0 irq=10520964/10520964, risc=36e9003c, bits: VSYNC HSYNC RISCI
[sck@house ~]$

Not sure, it will help identify the issue, but...

Scott

sck_nogas · Post by **sck_nogas** » Wed Dec 23, 2009 8:55 pm

To identify my system further, I compiled ZoneMinder 1.24.2 from source, though I'm thinking of pulling the SVN repo to see if there's a change.

The other solution I have to detect and fix the issue is to do something like this..

Code: Select all

#!/bin/bash

devices="video0 video1 video2 video3"

while true ; do 
	for device in $devices ; do
		VIDEO=$(cat /proc/sed/root/dev/$device 2>&1) 
		errcode=$?
		if [ "$errcode" -eq "255" ]; then  
			echo " Video Device $device is dead, restarting ZoneMinder"
			service zm stop; modprobe -r bttv; sleep 1; modprobe  bttv; service zm start
		fi
	done
	sleep 30
done

But that is such a hack...

Scott

Post by **cordel** » Fri Dec 25, 2009 6:50 am

Check dmesg and /proc/interrupts for problems and conflicts.
Some MB bios does not play nice with BT878 devices, you might have to replace your MB again worst case.

sck_nogas · Post by **sck_nogas** » Sat Dec 26, 2009 4:16 pm

Well, my `dmesg` output just contains alot of...

Code: Select all

[sck@house ~]$ dmesg | tail
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba501c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba501c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
bttv2: reset, reinitialize
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
[sck@house ~]$ dmesg | wc -l
2369
[sck@house ~]$ dmesg | sort -u 
8636363).
bttv2: PLL can sleep, using XTAL (28636363).
bttv2: reset, reinitialize
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba501c, bits: VSYNC HSYNC RISCI
bttv2: timeout: drop=0 irq=18645223/18645223, risc=10ba503c, bits: VSYNC HSYNC RISCI
[sck@house ~]$

And my script hasn't triggered yet.

Likewise, the /proc.interrupts shows nothing... yet...

Code: Select all

[sck@house ~]$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
  0:  313366768          0          0          0    IO-APIC-edge  timer
  1:          8          0          0          0    IO-APIC-edge  i8042
  4:         10          0          0          0    IO-APIC-edge  serial
  6:          6          0          0          0    IO-APIC-edge  floppy
  7:          1          0          0          0    IO-APIC-edge  parport0
  8:          3          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 12:        126          0          0          0    IO-APIC-edge  i8042
 50:    7712819          0   20946691          0   IO-APIC-level  bttv0
 58:   10673260          0   17494829          0   IO-APIC-level  bttv1
 66:    2908030          0   22055595          0   IO-APIC-level  bttv2
 74:    6951547          0   22264687          0   IO-APIC-level  bttv3
 82:       1571          0          0   56651381         PCI-MSI  eth0
201:          2          0          0          0   IO-APIC-level  ehci_hcd:usb1
209:     197328          0      38194          0   IO-APIC-level  ohci_hcd:usb2
225:      84503     145656    9449494          0         PCI-MSI  ahci
NMI:          0          0          0          0 
LOC:  313355262  313355258  313355470  313355466 
ERR:          1
MIS:          0
[sck@house ~]$

I'll keep digging...

Thanks for the assistance,

Scott

sck_nogas · Post by **sck_nogas** » Wed Jan 06, 2010 3:36 am

Okay, we have an update!!

My Shell code failed to detect since the error code was always -1 so I wrote an ugly perl version...

Code: Select all

#!/usr/bin/perl

@devices=qw(video0 video1 video2 video3);
$count =2;
while ($count>1) {
	foreach $device(@devices) {
		open (VIDEO, "cat /proc/self/root/dev/$device 2>&1 |");
		while (<VIDEO>){ 
			$line=$_;
			if ($line =~ /Input\/output error/){
				print " Video Device $device is dead, restarting ZoneMinder\n";
				system("service zm stop; modprobe -r bttv; sleep 5; modprobe  bttv; service zm start");
			}
		}
	}
	sleep 30;
}

But that still failed to catch this, so... I looked into this further with /proc/interrupts and dmesg and saw this...

Code: Select all

[sck@house ~]$ sudo cat /proc/self/root/dev/video[0-3]
cat: /proc/self/root/dev/video0: Device or resource busy
cat: /proc/self/root/dev/video1: Device or resource busy
cat: /proc/self/root/dev/video2: Input/output error
cat: /proc/self/root/dev/video3: Input/output error
sck@house ~]$ cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
  0: 1217368931          0          0          0    IO-APIC-edge  timer
  1:          8          0          0          0    IO-APIC-edge  i8042
  4:         10          0          0          0    IO-APIC-edge  serial
  6:          6          0          0          0    IO-APIC-edge  floppy
  7:          1          0          0          0    IO-APIC-edge  parport0
  8:          3          0          0          0    IO-APIC-edge  rtc
  9:          0          0          0          0   IO-APIC-level  acpi
 12:        126          0          0          0    IO-APIC-edge  i8042
 50:   12517132          0   95613228          0   IO-APIC-level  bttv0
 58:   21938330          0   31403216          0   IO-APIC-level  bttv1
 66:    2908030          0   22055595          0   IO-APIC-level  bttv2
 74:   13706973          0  171638751          0   IO-APIC-level  bttv3
 82:       1571          0          0  445758629         PCI-MSI  eth0
201:          2          0          0          0   IO-APIC-level  ehci_hcd:usb1
209:     695908          0     217467          0   IO-APIC-level  ohci_hcd:usb2
225:    1806795     172572   43149540          0         PCI-MSI  ahci
NMI:          0          0          0          0 
LOC: 1217325207 1217325204 1217325812 1217325806 
ERR:          1
MIS:          0
[sck@house ~]$ dmesg 
bttv3: unloading
bttv2: unloading
bttv1: unloading
bttv0: unloading
Linux video capture interface: v2.00
bttv: driver version 0.9.16 loaded
bttv: using 8 buffers with 2080k (520 pages) each for capture
bttv: Bt8xx card found (0).
ACPI: PCI Interrupt 0000:02:08.0[A] -> Link [LNKB] -> GSI 19 (level, low) -> IRQ 50
bttv0: Bt878 (rev 17) at 0000:02:08.0, irq: 50, latency: 64, mmio: 0xfbfff000
bttv0: detected: Provideo PV150A-1 [card=98], PCI subsystem ID is aa00:1460
bttv0: using: ProVideo PV150 [card=98,autodetected]
bttv0: gpio: en=00000000, out=00000000 in=00ffffff [init]
bttv0: using tuner=-1
bttv0: i2c: checking for TDA9875 @ 0xb0... not found
bttv0: i2c: checking for TDA7432 @ 0x8a... not found
bttv0: i2c: checking for TDA9887 @ 0x86... not found
bttv0: registered device video0
bttv0: registered device vbi0
bttv0: PLL: 28636363 => 35468950 . ok
bttv: Bt8xx card found (1).
ACPI: PCI Interrupt 0000:02:09.0[A] -> Link [LNKC] -> GSI 18 (level, low) -> IRQ 58
bttv1: Bt878 (rev 17) at 0000:02:09.0, irq: 58, latency: 64, mmio: 0xfbffd000
bttv1: detected: Provideo PV150A-2 [card=98], PCI subsystem ID is aa01:1461
bttv1: using: ProVideo PV150 [card=98,autodetected]
bttv1: gpio: en=00000000, out=00000000 in=00ffffff [init]
bttv1: using tuner=-1
bttv1: i2c: checking for TDA9875 @ 0xb0... not found
bttv1: i2c: checking for TDA7432 @ 0x8a... not found
bttv1: i2c: checking for TDA9887 @ 0x86... not found
bttv1: registered device video1
bttv1: registered device vbi1
bttv1: PLL: 28636363 => 35468950 . ok
bttv: Bt8xx card found (2).
ACPI: PCI Interrupt 0000:02:0a.0[A] -> Link [LNKD] -> GSI 17 (level, low) -> IRQ 66
bttv2: Bt878 (rev 17) at 0000:02:0a.0, irq: 66, latency: 64, mmio: 0xfbffb000
bttv2: detected: Provideo PV150A-3 [card=98], PCI subsystem ID is aa02:1462
bttv2: using: ProVideo PV150 [card=98,autodetected]
bttv2: gpio: en=00000000, out=00000000 in=00ffffff [init]
bttv2: using tuner=-1
bttv2: i2c: checking for TDA9875 @ 0xb0... not found
bttv2: i2c: checking for TDA7432 @ 0x8a... not found
bttv2: i2c: checking for TDA9887 @ 0x86... not found
bttv2: registered device video2
bttv2: registered device vbi2
bttv2: PLL: 28636363 => 35468950 . ok
bttv: Bt8xx card found (3).
ACPI: PCI Interrupt 0000:02:0b.0[A] -> Link [LNKA] -> GSI 16 (level, low) -> IRQ 74
bttv3: Bt878 (rev 17) at 0000:02:0b.0, irq: 74, latency: 64, mmio: 0xfbff9000
bttv3: detected: Provideo PV150A-4 [card=98], PCI subsystem ID is aa03:1463
bttv3: using: ProVideo PV150 [card=98,autodetected]
bttv3: gpio: en=00000000, out=00000000 in=00ffffff [init]
bttv3: using tuner=-1
bttv3: i2c: checking for TDA9875 @ 0xb0... not found
bttv3: i2c: checking for TDA7432 @ 0x8a... not found
bttv3: i2c: checking for TDA9887 @ 0x86... not found
bttv3: registered device video3
bttv3: registered device vbi3
bttv3: PLL: 28636363 => 35468950 . ok
irq 74: nobody cared (try booting with the "irqpoll" option)
 [<c044d1ea>] __report_bad_irq+0x2b/0x69
 [<c044d3d7>] note_interrupt+0x1af/0x1e8
 [<c044c9e5>] handle_IRQ_event+0x45/0x8c
 [<c044cac7>] __do_IRQ+0x9b/0xd6
 [<c044ca2c>] __do_IRQ+0x0/0xd6
 [<c040749e>] do_IRQ+0x99/0xc3
 [<c0405946>] common_interrupt+0x1a/0x20
 [<c0403ce7>] mwait_idle+0x25/0x38
 [<c0403ca8>] cpu_idle+0x9f/0xb9
 =======================
handlers:
[<f8b67bfa>] (bttv_irq+0x0/0x74a [bttv])
Disabling IRQ #74

So I added the irqpoll statement to the kernel line of my /boot/grub/grub.conf and rebooted.

We'll see if that fixes the problem...

Scott

sck_nogas · Post by **sck_nogas** » Wed Jan 20, 2010 11:34 pm

Well it's been several weeks now, and it seems to have fixed the issue.

So, if you are having this issue, use the IRQPOLL kernel parameter and this problem should go away once you reboot into the new kernel.

Scott