need to debug segfault in zmc 1.23.3

[chris_at_1006_dot_org]

Hi,

I'm running a zm 1.23.3 compiled from source on an Ubuntu 8.04 server.

The problem is I get a lot of seg faults of the following kind:

Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Got signal (Segmentation fault), crashing]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Signal address is 0xdf9027dc, from 0x805f1e1 ]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc [0x80602d1]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc [0x8061b6d]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc [0x8065eba]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc [0x804afbd]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe0) [0xb7a78450]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: ERR [Backtrace: /opt/zm/bin/zmc(__gxx_personality_v0+0x20d) [0x804a821]]
Feb 21 18:01:29 xxxxxxx zmc_m3[24744]: INF [Backtrace complete]
Feb 21 18:01:29 xxxxxxx zmdc[24511]: ERR ['zmc -m 3' exited abnormally, exit status 11]

I'd like to debug this somehow, but are a bit stuck.
Although I enabled core dumps (with ulimit -c 1000000 in /etc/profile) I don't
get any from zmc's crashes. This is because they're trapped somehow and dealt with
inside zmc, right?

If I attach gdb to a zmc process PID while it's still alive I get

Feb 21 21:26:59 xxxxxxx zmdc[25679]: INF ['zmc -m 2' crashed, signal 8]
Feb 21 21:26:59 xxxxxxx zmwatch[25729]: INF ['zmc -m 2' stopping at 09/02/21 21:25:14]
Feb 21 21:26:59 xxxxxxx zmdc[25679]: INF [Starting pending process, zmc -m 2]

So again zmc is escaping me...

Could somebody more familiar with ZM please provide a hint how I could investigate
the above seg fault? Any hints welcome.

Please note I'm reasonably fit with C, the gcc toolchain and the posix environment
but I don't know the inner workings of ZM...

Bye,
Chris.

[nuck]

See this - http://www.zoneminder.com/forums/viewtopic.php?t=13051 , specifically Phil's post - Posted: Tue Feb 10, 2009 5:05 pm . It should help you out some.

[chris_at_1006_dot_org]

Thank you nuck,

following Phil's tip (addr2line) I could pinpoint the exact point
where it segfaults: in line 188 of zm_remote_camera.cpp!
Below are 3 stacktraces on 2 different boxes that all point to this line.

184 int RemoteCamera::ReadData( Buffer &buffer, int bytes_expected )
185 {
186 fd_set rfds;
187 FD_ZERO(&rfds);
188 FD_SET(sd, &rfds); // <- HERE!
189

The video guys tell me it happens more likely when there is "action": alerts triggered etc...

In case any of the devs is reading: do you have any idea what could
be causing this? Some inconsistent settings? Resource starvation?

Thanks and Bye,
Chris.

Mar 2 17:29:50 box_a zmc_m3[5646]: INF [Debug Level = 0, Debug Log = <none>]
Mar 2 17:29:50 box_a zmc_m3[5646]: INF [Starting Capture]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Got signal (Segmentation fault), crashing]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Signal address is 0xdff04a3c, from 0x805f1e1 ]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc [0x80602d1]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc [0x8061b6d]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc [0x8065eba]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc [0x804afbd]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe0) [0xb7b3b450]]
Mar 2 17:29:50 box_a zmc_m3[5646]: ERR [Backtrace: /opt/zm/bin/zmc(__gxx_personality_v0+0x20d) [0x804a821]]
Mar 2 17:29:50 box_a zmc_m3[5646]: INF [Backtrace complete]

user@box_a:/var/log$ addr2line -e /opt/zm/bin/zmc 0x805f1e1 0x80602d1 0x8061b6d 0x8065eba 0x804afbd
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:188
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:940
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:1094
/home/user/ZoneMinder-1.23.3/src/zm_monitor.cpp:2124
/home/user/ZoneMinder-1.23.3/src/zmc.cpp:250

---

Mar 2 17:30:49 box_a zmc_m1[5666]: INF [Debug Level = 0, Debug Log = <none>]
Mar 2 17:30:49 box_a zmc_m1[5666]: INF [Starting Capture]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Got signal (Segmentation fault), crashing]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Signal address is 0xdf8b33ec, from 0x805f1e1 ]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc [0x8060384]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc [0x8061b6d]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc [0x8065eba]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc [0x804afbd]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe0) [0xb7b53450]]
Mar 2 17:30:49 box_a zmc_m1[5666]: ERR [Backtrace: /opt/zm/bin/zmc(__gxx_personality_v0+0x20d) [0x804a821]]
Mar 2 17:30:49 box_a zmc_m1[5666]: INF [Backtrace complete]

user@box_a:/var/log$ addr2line -e /opt/zm/bin/zmc 0x805f1e1 0x8060384 0x8061b6d 0x8065eba 0x804afbd
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:188
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:628
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:1094
/home/user/ZoneMinder-1.23.3/src/zm_monitor.cpp:2124
/home/user/ZoneMinder-1.23.3/src/zmc.cpp:250

---

Mar 5 07:58:59 box_b zmc_m3[23375]: INF [Debug Level = 0, Debug Log = <none>]
Mar 5 07:58:59 box_b zmc_m3[23375]: INF [Starting Capture]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Got signal (Segmentation fault), crashing]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Signal address is 0xdfc07f3c, from 0x805f1e1 ]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc [0x805f1e1]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc [0x8060384]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc [0x8061b6d]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc [0x8065eba]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc [0x804afbd]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe0) [0xb7ab7450]]
Mar 5 07:58:59 box_b zmc_m3[23375]: ERR [Backtrace: /opt/zm/bin/zmc(__gxx_personality_v0+0x20d) [0x804a821]]
Mar 5 07:58:59 box_b zmc_m3[23375]: INF [Backtrace complete]

user@box_b:~$ addr2line -e /opt/zm/bin/zmc 0x805f1e1 0x8060384 0x8061b6d 0x8065eba 0x804afbd
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:188
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:628
/home/user/ZoneMinder-1.23.3/src/zm_remote_camera.cpp:1094
/home/user/ZoneMinder-1.23.3/src/zm_monitor.cpp:2124
/home/user/ZoneMinder-1.23.3/src/zmc.cpp:250