Page 1 of 1

Instability

Posted: Wed Feb 01, 2012 1:04 am
by keyboardgnome
Howdy all,

so, after months and months and months of stable and reliable performance from my zoneminder 1.24.2 box (last silly problem was june 2011, but realistic stability since 2010), I've just started having a random problem creep up in the last few days that I wanted to see if anyone else has any experience with (I think it may be a failing capture card).

For the curious (but not necessary), the specs of the system are in this thread: http://www.zoneminder.com/forums/viewto ... 94&p=65557

Nevertheless, I have a PV-155 driving 6 cameras right now. What is happening now is any of the following at random intervals and sometimes can only be fixed by a reboot or hitting F5 for a refresh: that the images are jumping from one camera to another (reminds me of the old interlace issue), the images are all black, or the image is the vertical bar color tone test.

One thing I don't like is that I don't know how to trigger these random bits of failure other than just wait a few minutes after a reboot. If I reboot, the system is fine, but for a short period of time.

I have a LOT of "zm_debug" logs, that all conclude with the following:

01/31/12 20:00:00.576038 zms[-1].DB3-zm_stream.cpp/143 [Send image width = 768, height = 480]
01/31/12 20:00:00.576061 zms[-1].DB3-zm_stream.cpp/146 [Last send image width = 768, height = 480]
01/31/12 20:00:00.576083 zms[-1].DB3-zm_stream.cpp/164 [Real image width = 768, height = 480]
01/31/12 20:00:00.690951 zms[-1].WAR-zm_monitor.cpp/3564 [Unable to store frame as shared memory invalid]
01/31/12 20:00:10.724042 zms[-1].ERR-zm_monitor.cpp/3579 [Terminating, last frame sent time 10.050136 secs more than maximum of 10.000000]
01/31/12 20:00:10.724188 zms[-1].DB1-zm_monitor.cpp/3588 [Cleaning swap files from /tmp/zmswap-m3/zmswap-q417094]



from zmdc.log
01/31/12 19:58:23.011378 zmdc[1142].INF [Starting pending process, zmc -d /dev/video1]
01/31/12 19:58:23.013529 zmdc[1142].INF ['zmc -d /dev/video1' starting at 12/01/31 19:58:23, pid = 19094]
01/31/12 20:00:00.693351 zmdc[1142].ERR ['zmc -d /dev/video1' exited abnormally, exit status 255]
01/31/12 20:00:23.546954 zmdc[1142].ERR ['zmc -d /dev/video2' exited abnormally, exit status 255]
01/31/12 20:00:23.547796 zmdc[1142].INF [Starting pending process, zmc -d /dev/video2]
01/31/12 20:00:23.549912 zmdc[1142].INF ['zmc -d /dev/video2' starting at 12/01/31 20:00:23, pid = 20181]
01/31/12 20:00:37.984030 zmdc[1142].ERR ['zmc -d /dev/video0' exited abnormally, exit status 255]
01/31/12 20:01:17.070901 zmdc[1142].INF [Starting pending process, zmc -d /dev/video0]
01/31/12 20:01:17.073116 zmdc[1142].INF ['zmc -d /dev/video0' starting at 12/01/31 20:01:17, pid = 20565]
01/31/12 20:01:20.080884 zmdc[1142].INF [Starting pending process, zmc -d /dev/v
ideo1]



Some additional relevant info that I deviate from normally:
kernel.shmall = 1073741824
kernel.shmmax = 1073741824

Thanks- let me know what else ya'll need to help look into this.

Re: Instability

Posted: Wed Feb 01, 2012 11:52 pm
by bb99
Sure reads like overheating the chips on the capture card. When was the last time you blew the case out, anyway after you do, put heatsinks on the four 878 chips. Could also be as easy as turning multi buffers off and setting the captures per frame value to 2 because of the doubled up cameras on at least one chip but I'm guessing 2 chips.

Re: Instability

Posted: Thu Feb 02, 2012 12:25 pm
by keyboardgnome
bb99 wrote:Sure reads like overheating the chips on the capture card. When was the last time you blew the case out, anyway after you do, put heatsinks on the four 878 chips. Could also be as easy as turning multi buffers off and setting the captures per frame value to 2 because of the doubled up cameras on at least one chip but I'm guessing 2 chips.
Thanks for the info-

The server is in a datacenter where the air is cleaned. I opened it up anyways earlier to make sure, and all was clean on the inside. I also gave it a chance to "cool off" before turning back on. Once I turned it back on, the problem came back after a minute or so. The other extra data point is that the system had a hard lockup for the first time last night; which tells me hardware is failing somewhere.

I'm prepping to make some purchases to replace the card.

Has anyone had any luck with a setup using two PV-149's? I'll probably throw this in another thread if I don't see a response.

For a long time, it's been chugging along happily at 5 fps per camera (I've never been able to get it to go faster). If I have to down clock it to two, I'd rather buy replacement cards as this system is performing a security monitoring function and a lot of stuff can be missed in a 2 fps capture (I'm not really happy with 5 fps either).