ZM hard crashing my system and other issues

[mklein]

Have built out a 2.4Ghz quadcore 4gb Ubuntu 64-bit 8.10 server with a BC 8-port capture card and hdhomerun.

Backend runs myth, mysql, asterisk, zm, ...the usual suspects.

After running for 10mins or so I can't ping box and local keybd/mouse access don't work...it is frozen.

I isolated this to the zm processes that get created. Box will stream hd ota via mythtv for 20hrs straight...as long as zm isn't running.

I also noticed that I end up with 100s? of apache instances at some point and then apache is unresponsive until I restart apache2 process.

Perhaps my math/reading were wrong in calculating shared memory. I have 7 inputs running at 640x480. Have the following settings from kernel:

Code: Select all

kernel.shmall=134217728
kernel.shmmax=134217728

Following is a sample camera entry in dmesg:

Code: Select all

[   19.255727] bttv: Bt8xx card found (7).
[   19.255738] bttv 0000:07:0b.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[   19.255749] bttv7: Bt878 (rev 17) at 0000:07:0b.0, irq: 16, latency: 32, mmio: 0xeb30e000
[   19.255760] bttv7: subsystem: 1836:1540 (UNKNOWN)
[   19.255761] please mail id, board name and the correct card= insmod option to video4linux-list@redhat.com
[   19.255763] bttv7: using:  *** UNKNOWN/GENERIC ***  [card=0,autodetected]
[   19.255788] bttv7: gpio: en=00000000, out=00000000 in=00ffffff [init]
[   19.290995] bttv7: tuner type unset
[   19.290998] bttv7: i2c: checking for MSP34xx @ 0x80... not found
[   19.291747] bttv7: i2c: checking for TDA9875 @ 0xb0... not found
[   19.292427] bttv7: i2c: checking for TDA7432 @ 0x8a... not found
[   19.293198] bttv7: registered device video7
[   19.293236] bttv7: registered device vbi7

I notice that when running mythzmserver? that images come across in blue tint.

Also...when I built ffmpeg from source...I only used swscale? option...not sharedmem or other stuff...perhaps this is cause?

Also...if I could get confirm that there's no way to get audio working from audio inputs under Ubuntu this w/be appreciated.

thanks in advance...

[mklein]

$#@&...now it appears my db is toast.

Is repair really an option here? I guess I need to start backing up mysql while getting zm crash-free.

I followed instr in sticky? post on building latest zm. Ended up building ffmpeg with more options than before and minor zm tweaks (grabbed from svn). Monitors weren't set to mocord like last time...merely monitors...still crashed.

I noticed when I switched to mpg streaming from jpg (after testing cambozola which worked fine) that I started getting kernel errors from bttv driver re:

Code: Select all

Feb 19 13:13:49 father kernel: [ 1112.215665] bttv1: OCERR @ 37864000,bits: HSYNC OFLOW FDSR OCERR*
Feb 19 13:13:49 father kernel: [ 1112.482594] bttv1: OCERR @ 37864000,bits: HSYNC OFLOW FDSR OCERR*

Shortly thereafter (even after setting back to jpg) I got kernel panic.

My shared mem settings for 7 640x480 cameras is:

Code: Select all

kernel.shmall=262144
kernel.shmmax=1252698795

Feb 19 13:21:31 father mysqld[6103]: 090219 13:21:31 [Note] /usr/sbin/mysqld: ready for connections.
Feb 19 13:21:31 father mysqld[6103]: Version: '5.0.67-0ubuntu6'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
Feb 19 13:21:36 father /etc/mysql/debian-start[6205]: Upgrading MySQL tables if necessary.
Feb 19 13:21:36 father /etc/mysql/debian-start[6213]: Looking for 'mysql' in: /usr/bin/mysql
Feb 19 13:21:36 father /etc/mysql/debian-start[6213]: Looking for 'mysqlcheck' in: /usr/bin/mysqlcheck
Feb 19 13:21:36 father mysqld[6103]: 090219 13:21:33 [ERROR] /usr/sbin/mysqld: Table './zm/Config' is marked as crashed and should be repaired
Feb 19 13:21:36 father /etc/mysql/debian-start[6213]: This installation of MySQL is already upgraded to 5.0.67, use --force if you still need to run mysql_upgrade
Feb 19 13:21:36 father mysqld[6103]: 090219 13:21:33 [Warning] Checking table:   './zm/Config'
Feb 19 13:21:37 father /etc/mysql/debian-start[6281]: Checking for insecure root accounts.
Feb 19 13:21:37 father /etc/mysql/debian-start[6290]: Triggering myisam-recover for all MyISAM tables
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:37 [ERROR] /usr/sbin/mysqld: Table './mythconverg/mythweb_sessions' is marked as crashed and should be repaired
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:37 [Warning] Checking table:   './mythconverg/mythweb_sessions'
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:37 [ERROR] /usr/sbin/mysqld: Table './mythconverg/program' is marked as crashed and should be repaired
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:37 [Warning] Checking table:   './mythconverg/program'
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:37 [ERROR] /usr/sbin/mysqld: Table './mythconverg/settings' is marked as crashed and should be repaired
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:37 [Warning] Checking table:   './mythconverg/settings'
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:38 [ERROR] /usr/sbin/mysqld: Table './mythconverg/upnpmedia' is marked as crashed and should be repaired
Feb 19 13:21:40 father mysqld[6103]: 090219 13:21:38 [Warning] Checking table:   './mythconverg/upnpmedia'
Feb 19 13:21:46 father x-session-manager[6312]: WARNING: Unable to find provider 'gnome-wm' of required component 'windowmanager'
Feb 19 13:21:55 father x-session-manager[6312]: WARNING: Application 'gnome-wm.desktop' failed to register before timeout
Feb 19 13:21:55 father lircd-0.8.3[6856]: lircd(userspace) ready
Feb 19 13:22:07 father x-session-manager[6312]: WARNING: Application 'libcanberra-login-sound.desktop' failed to register before timeout
Feb 19 13:22:11 father x-session-manager[6312]: WARNING: Could not launch application 'nm-applet.desktop': Unable to start application: Failed to execute child process "nm-applet"
(No such file or directory)

[mklein]

No longer booting X. Makes no difference.

Was getting 'increase ring buffer' messages to I boosted frames from 40 to 80...no difference.

Code: Select all

Feb 19 14:37:38 father zma_m4[22829]: WAR [Approaching buffer overrun, consider slowing capture, simplifying analysis or increasing ring buffer size]

My zm logs show constant restarting of capture and analysis daemons.

Code: Select all

02/19/2009 14:44:02.236696 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk1, time since last analysis 9 seconds (1235083442-1235083433)]
02/19/2009 14:44:13.743434 zmwatch[22205].INF ['zma -m 2' stopping at 09/02/19 14:44:13]
02/19/2009 14:45:13.756061 zmwatch[22205].INF [Restarting capture daemon for FrontDoor, time since last capture 8 seconds (1235083513-1235083505)]
02/19/2009 14:45:19.844267 zmwatch[22205].INF ['zmc -d /dev/video0' stopping at 09/02/19 14:45:18]
02/19/2009 14:46:01.144346 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk2, time since last analysis 6 seconds (1235083561-1235083555)]
02/19/2009 14:46:16.959744 zmwatch[22205].INF ['zma -m 3' stopping at 09/02/19 14:46:15]
02/19/2009 14:46:58.099412 zmwatch[22205].INF [Restarting capture daemon for FrontDoor, time since last capture 9 seconds (1235083618-1235083609)]
02/19/2009 14:47:10.554982 zmwatch[22205].INF ['zmc -d /dev/video0' stopping at 09/02/19 14:47:08]
02/19/2009 14:47:57.144152 zmwatch[22205].INF [Restarting analysis daemon for FrontDoor, time since last analysis 12 seconds (1235083677-1235083665)]
02/19/2009 14:48:25.611326 zmwatch[22205].INF ['zma -m 1' stopping at 09/02/19 14:48:24]
02/19/2009 14:48:47.834185 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk3, time since last analysis 8 seconds (1235083727-1235083719)]
02/19/2009 14:48:59.959161 zmwatch[22205].INF ['zma -m 4' stopping at 09/02/19 14:48:55]
02/19/2009 14:49:41.937775 zmwatch[22205].INF [Restarting capture daemon for FrontDoor, time since last capture 13 seconds (1235083781-1235083768)]
02/19/2009 14:49:46.911750 zmwatch[22205].INF ['zmc -d /dev/video0' stopping at 09/02/19 14:49:46]
02/19/2009 14:50:16.918639 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk2, time since last analysis 6 seconds (1235083816-1235083810)]
02/19/2009 14:50:37.888323 zmwatch[22205].INF ['zma -m 3' stopping at 09/02/19 14:50:35]
02/19/2009 14:50:48.531604 zmwatch[22205].INF [Restarting analysis daemon for FrontDoor, time since last analysis 11 seconds (1235083848-1235083837)]
02/19/2009 14:51:11.576871 zmwatch[22205].INF ['zma -m 1' stopping at 09/02/19 14:50:56]
02/19/2009 14:52:32.651017 zmwatch[22205].INF [Restarting capture daemon for Sidewalk1, time since last capture 14 seconds (1235083952-1235083938)]
02/19/2009 14:53:49.948007 zmwatch[22205].INF ['zmc -d /dev/video1' stopping at 09/02/19 14:53:46]
02/19/2009 14:54:10.445227 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk3, time since last analysis 8 seconds (1235084050-1235084042)]
02/19/2009 14:54:24.448903 zmwatch[22205].INF ['zma -m 4' stopping at 09/02/19 14:54:24]
02/19/2009 14:54:44.452689 zmwatch[22205].INF [Restarting analysis daemon for FrontDoor, time since last analysis 6 seconds (1235084084-1235084078)]
02/19/2009 14:55:00.177951 zmwatch[22205].INF ['zma -m 1' stopping at 09/02/19 14:54:49]
02/19/2009 14:55:24.300749 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk2, time since last analysis 7 seconds (1235084124-1235084117)]
02/19/2009 14:56:07.115630 zmwatch[22205].INF ['zma -m 3' stopping at 09/02/19 14:55:35]
02/19/2009 14:56:26.418231 zmwatch[22205].INF [Restarting analysis daemon for FrontDoor, time since last analysis 7 seconds (1235084186-1235084179)]
02/19/2009 14:56:35.971651 zmwatch[22205].INF ['zma -m 1' stopping at 09/02/19 14:56:31]
02/19/2009 14:56:55.978561 zmwatch[22205].INF [Restarting capture daemon for Sidewalk2, time since last capture 6 seconds (1235084215-1235084209)]
02/19/2009 14:57:04.536672 zmwatch[22205].INF ['zmc -d /dev/video2' stopping at 09/02/19 14:57:03]
02/19/2009 14:57:15.101850 zmwatch[22205].INF [Restarting analysis daemon for Sidewalk2, time since last analysis 7 seconds (1235084235-1235084228)]
02/19/2009 14:57:21.612944 zmwatch[22205].INF ['zma -m 3' stopping at 09/02/19 14:57:19]
02/19/2009 14:57:41.618283 zmwatch[22205].INF [Restarting capture daemon for Driveway, time since last capture 6 seconds (1235084261-1235084255)]
02/19/2009 14:57:53.213049 zmwatch[22205].INF ['zmc -d /dev/video5' stopping at 09/02/19 14:57:50]
02/19/2009 14:58:25.354349 zmwatch[22205].INF [Restarting capture daemon for Sidewalk3, time since last capture 11 seconds (1235084305-1235084294)]

Also...zm seems to be heavily pegging my CPUs:

Code: Select all

 6464 www-data  20   0  238m  77m  73m R   59  1.9   1:36.67 zma
 7380 www-data  20   0  220m  59m  55m R   57  1.5   0:12.26 zma
 6470 www-data  20   0  238m  77m  73m R   53  1.9   1:41.74 zma
 6476 www-data  20   0  239m  78m  73m R   51  2.0   1:41.29 zma
 7219 www-data  20   0  220m  59m  55m R   48  1.5   0:51.35 zma
 6496 www-data  20   0  221m  60m  55m R   46  1.5   1:08.54 zma
 6458 www-data  20   0  220m  59m  55m R   41  1.5   1:47.91 zma
 7142 www-data  20   0  242m  80m  76m S    6  2.0   0:09.30 zmc
 6456 www-data  20   0  224m  63m  59m S    5  1.6   0:11.90 zmc
 6467 www-data  20   0  242m  80m  76m S    5  2.0   0:11.38 zmc
 6479 www-data  20   0  224m  63m  59m S    5  1.6   0:10.58 zmc
 6487 www-data  20   0  224m  63m  59m S    5  1.6   0:10.78 zmc
 7397 www-data  20   0  242m  80m  76m S    5  2.0   0:00.68 zmc
 7184 www-data  20   0  224m  63m  59m S    5  1.6   0:06.02 zmc

Due to frequent restarts I am missing capture/analysis for 10s of seconds at a time per camera.

Hard-crashing is my primary problem...zm is useless for me along with bc card until I get this fixed. Please assist. Wife will kill me if I don't get something stable sometime soon.

Also...issuing "repair table <foo>" in mysql for each mangled table seemed to please mysql.

I haven't installed heat sinks yet but card is only running for 10mins before system crashes.

[nuck]

If you are thinking heat may be an issue why not back off on the system load some and see what happens? Perhaps start out with one monitor enabled @ 320x240 and go from there?

[mklein]

Not sure it's heat really...just mentioning it.

I'm in a cool room and case is extremely ventilated and numerous fans.

Good idea...I'm running memtest to check for some occasional swap errors I saw and after 24hrs or so will bring up system with single cam.

[mklein]

After several hours...memtest is clean. I was seeing occasional/rare swap messages so this is generally mentioned to run as test.

I had great difficulty removing cams while zm was running. Took 30-60secs a camera and many times this delete stalled out.

I could see nothing in apache/zm logs why this was timing out...my guess is due to process killing not occuring...when I tried to kill certain zm camera processes too it just hangs and I see zm log entries regarding "stopping/restarting" daemons.

Easiest route wasn't to use ui but to stop zm, drop zm, recreate zm, and now a single camera.

I will run single mocord camera at 640x480 and 30fps/10fps with ntsc m and yuv420.

I will keep shmem settings at their current LARGE setting (from this thread).

Will keep everybody posted...

[mklein]

Me thinks mocord was killing me...7 cams, 640x480 @ 30/10 fps...now on modect.

It took positively forever to remove old event jpgs under webroot.

Hasn't hung yet (after 15? mins) and with one cam I'm seeing saner logs/etc.

Events logs are minimal and predictable.

I may see if 24-bit palette works...using yuv420 and read there was performance penalty...won't be wanting that if at all possible.

[nuck]

May I suggest reading - http://www.zoneminder.com/wiki/index.ph ... uch_CPU.3F ??

Try setting fps @ 10 / 30 and see if you aren't any happier.

[nuck]

I may see if 24-bit palette works...using yuv420 and read there was performance penalty...won't be wanting that if at all possible.

There is a performance penalty for everything you throw at your system. You'll want to back down the frame rate, colour and resolution wherever you can to get better overall system performance. If you've got a couple cameras that you can get by with lower resolution then by all means cut them down to size. Same with frame rates. And colour.

[mklein]

I'll be happy if I can best the performance of a chinese-make linux-based dvr which cost me $1K approx. No raid option and windows only software but it works tits for the most part. Missing some configurability but pretty much an appliance. I've had it for a year now and not ONCE had to do anything to it...and it's a 16-port with 12-ports active.

My own machine is still bogging down bad, not only is apache kicking it but other things (like a simple process list in a terminal window) don't finish completely. I believe this happened somewhere around 5-7 cameras.

I will scale down res...but this isn't what I hoped I would have to do with a quadcore 2.4Ghz 4gb 1600fsb.....

[nuck]

My own machine is still bogging down bad, not only is apache kicking it but other things (like a simple process list in a terminal window) don't finish completely. I believe this happened somewhere around 5-7 cameras.

I will scale down res...but this isn't what I hoped I would have to do with a quadcore 2.4Ghz 4gb 1600fsb.....

As Phil said to me about a week ago; Your cpu load seems high, and he referred me to that wiki page. Seriously, try cutting down the resolution and see where it gets you. I think you may be pleasantly surprised. And face it, unless you need to be able to count the freckles on someone's face, do you really need it?

All that said, I do remember Phil saying something in another thread sometime in the past few days about the bttv driver and it running into problems switching at high rates. Can't remember exactly what it was, but he may well chime in when he reads this.

Something else you may want to look into is ZM_V4L_MULTI_BUFFER _and_ ZM_CAPTURES_PER_FRAME .. In fact, do a search on either of those terms and I think you'll find what I mentioned Phil had said about the BT8xx cards.

[mklein]

Thanks for the tips.

If interlaced higher-res is really BS then ok. Analog cameras often suck anyways so I'm dealing with what I paid for ($150/camera roughly).

I need to start looking into forensic-quality (license plate/court proof) cameras that ideally have analog and ethernet interfaces...and are preferrable weatherproof themselves and don't require housing...and are zm best-of-class.

Perhaps when I cut the res down the blue tint to my mythzm screens will go away...

Letting myself and the computer cool down for the night...ciao!

[mklein]

Much better now...

I went to 320x240, 10/30 for monitor/capture and all cameras are on modect instead of mocord.

Cpu is infinitely less pegged:

Code: Select all

top - 16:25:01 up  2:01,  3 users,  load average: 0.12, 0.17, 0.14
Tasks: 191 total,   2 running, 189 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.5%us,  1.0%sy,  0.0%ni, 93.4%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   4054224k total,  2457320k used,  1596904k free,    35592k buffers
Swap: 11871992k total,     5500k used, 11866492k free,  1688228k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6448 www-data  20   0  175m  14m  11m S    5  0.4   3:20.12 zma
22351 munin     20   0 49620 7532 2240 S    3  0.2   0:00.08 munin-update
 6454 www-data  20   0  174m  13m  11m S    2  0.3   3:05.97 zma
 6442 www-data  20   0  174m  13m  11m S    2  0.3   3:19.97 zma
 6466 www-data  20   0  174m  13m  11m S    2  0.3   1:57.48 zma
 6472 www-data  20   0  174m  13m  11m S    2  0.3   2:12.47 zma
 6460 www-data  20   0  174m  13m  11m S    1  0.3   2:25.77 zma
 6478 www-data  20   0  174m  13m  11m S    1  0.3   2:05.64 zma
 6416 www-data  20   0 65916 9452 1572 S    1  0.2   0:03.52 zmdc.pl
 6444 www-data  20   0  175m  14m  12m S    1  0.4   0:33.04 zmc
 6451 www-data  20   0  175m  14m  12m S    1  0.4   0:33.02 zmc
 6457 www-data  20   0  175m  14m  12m S    1  0.4   0:32.36 zmc
 6463 www-data  20   0  175m  14m  12m S    1  0.4   0:32.24 zmc
 6469 www-data  20   0  175m  14m  12m S    1  0.4   0:32.14 zmc
 7297 mklein    20   0  286m  23m  13m S    1  0.6   0:01.22 gnome-panel
22333 mklein    20   0 19100 1384  988 R    1  0.0   0:00.02 top
    1 root      20   0  4100  904  612 S    0  0.0   0:01.10 init

The zm web intf is working well (no special zones yet) and zm from mythtv is quite nice too...not a huge color tint anymore in mythzm videos but still a minor color tint.

Finally got HD OTA in mythtv smooth as a baby's bottom today too...woohoo!

Now just need to put heat sinks on and experiment/tweak more. Will recalc my shmem settings so I'm not on larger values as I'm now on 320x240.