ZM 1.24.2 on CentOS 5.4.
Both zma and zmc are crashing, zmc more often (on IP camera). The crashes do not seem to be related timewise.
Any suggestions?
-------------
zma:
Nov 25 06:38:32 xxxxxxx zma_m2[28078]: INF [Monitor-2: 178000 - Processing at 4.95 fps]
Nov 25 06:40:41 xxxxxxx zma_m2[31868]: INF [Monitor-2: 064 - Gone into alarm state]
Nov 25 06:40:41 xxxxxxx zma_m2[31868]: INF [Monitor-2: 064 - Opening new event 1102, alarm start]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: INF [Got signal 11 (Segmentation fault), crashing]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Signal address is 0xd963d0, from 0x5f3486]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /lib/libc.so.6 [0x5f3486]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /lib/libc.so.6 [0x5f3486]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /lib/libc.so.6(__libc_malloc+0x67) [0x5f4d87]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/lib/libstdc++.so.6(_Znwj+0x27) [0xbf2ab7]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x8056721]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x8056c0d]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x80572fd]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x805755e]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x8069ac0]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x8069cea]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x8069ebd]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x8070bee]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x804cca1]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /lib/libc.so.6(__libc_start_main+0xdc) [0x59ee9c]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: ERR [Backtrace: /usr/local/bin/zma [0x804c931]]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: INF [Backtrace complete, please execute the following command for more
information]
Nov 25 06:40:42 xxxxxxx zma_m2[31868]: INF [addr2line -e /usr/local/bin/zma 0x5f3486 0x5f3486 0x5f4d87 0xbf2a
b7 0x8056721 0x8056c0d 0x80572fd 0x805755e 0x8069ac0 0x8069cea 0x8069ebd 0x8070bee 0x804cca1 0x59ee9c 0x804c9
31]
# addr2line -e /usr/local/bin/zma 0x5f3486 0x5f3486 0x5f4d87 0xbf2ab7 0x8056721 0x8056c0d 0x80572fd 0x805755e 0x8069ac0 0x8069cea 0x8069ebd 0x8070bee 0x804cca1 0x59ee9c 0x804c931
??:0
??:0
??:0
??:0
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:358
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:369
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:821
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:927
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:949
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_map.h:420
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_map.h:348
/root/ZoneMinder-1.24.2/src/zm_monitor.cpp:1240
/root/ZoneMinder-1.24.2/src/zma.cpp:119
??:0
??:0
-------------
zmc (on IP camera):
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: INF [Got signal 11 (Segmentation fault), crashing]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Signal address is 0xdfaea37c, from 0x807a9a5]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x807a9a5]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x807a9a5]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x807badf]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x807d90a]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x806ecea]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x804d1e2]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /lib/libc.so.6(__libc_start_main+0xdc) [0x59ee9c]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: ERR [Backtrace: /usr/local/bin/zmc [0x804c961]]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: INF [Backtrace complete, please execute the following command for more
information]
Nov 25 06:00:40 xxxxxxx zmc_m2[31480]: INF [addr2line -e /usr/local/bin/zmc 0x807a9a5 0x807a9a5 0x807badf 0x8
07d90a 0x806ecea 0x804d1e2 0x59ee9c 0x804c961]
Nov 25 06:00:40 xxxxxxx zmdc[27814]: ERR ['zmc -m 2' exited abnormally, exit status 11]
# addr2line -e /usr/local/bin/zmc 0x807a9a5 0x807a9a5 0x807badf 0x807d90a 0x806ecea 0x804d1e2 0x59ee9c 0x804c961
/root/ZoneMinder-1.24.2/src/zm_remote_camera_http.cpp:134
/root/ZoneMinder-1.24.2/src/zm_remote_camera_http.cpp:134
/root/ZoneMinder-1.24.2/src/zm_remote_camera_http.cpp:887
/root/ZoneMinder-1.24.2/src/zm_remote_camera_http.cpp:1041
/root/ZoneMinder-1.24.2/src/zm_monitor.cpp:2474
/root/ZoneMinder-1.24.2/src/zmc.cpp:258
??:0
??:0
Thanks in advance!
Segmentation fault in zmc and zma
-
bristoldave
- Posts: 48
- Joined: Thu Mar 05, 2009 9:42 pm
I'm convinced this problem is related to a lot of the other seg fault problems with IP cameras reported on the forum, such as http://www.zoneminder.com/forums/viewtopic.php?t=14721 and http://www.zoneminder.com/forums/viewtopic.php?t=13340
A lot of these seem to be seperate to the shared memory issue. Certainly the problem I'm experiencing is.
Can any developers look into this please? It must be something simple, but I'm not familiar enough with the code!
A lot of these seem to be seperate to the shared memory issue. Certainly the problem I'm experiencing is.
Can any developers look into this please? It must be something simple, but I'm not familiar enough with the code!
This is what I was able to figure out:
--- zma
The program coredumps in the code related to linked monitors. I figured out I accidentally configured linked monitor for the one in troubles. When I removed it, everything went smoothly. I am leaving the rest to the developers.
--- zmc
The following patch fixed segmentation fault, but it is not the fix for the problem itself.
I believe when the camera returns zero length response, zma thinks the communication is lost, disconnects the file descriptor, and returns the code 0:
int RemoteCameraHttp::ReadData( Buffer &buffer, int bytes_expected )
{
....
Debug( 3, "Socket closed" );
Disconnect();
return( 0 );
....
}
I suspect this return code is not handled properly, and another attempt to read is performed on closed descriptor, which causes coredump. The patch allows to avoid it and finish the program properly instead. It is a hack, and I am leaving the rest to the developers.
##############################################################################
*** zm_remote_camera_http.cpp.orig 2009-11-26 14:30:38.000000000 -0600
--- zm_remote_camera_http.cpp 2009-11-26 14:12:00.000000000 -0600
***************
*** 129,134 ****
--- 129,140 ----
int RemoteCameraHttp::ReadData( Buffer &buffer, int bytes_expected )
{
+ if(sd < 0)
+ {
+ Error( "Bad descriptor" );
+ return -1;
+ }
+
fd_set rfds;
FD_ZERO(&rfds);
FD_SET(sd, &rfds);
##############################################################################
--- MySQL
I improved package stability with the following patch. Sometimes, when there is not enough activity, the connection to MySQL times out, and the program needs to restart. In case of zma it may cause the lost of an event, which occured after long period of inactivity. This patch allows automatic reconnection to MySQL. You must run MySQL version 5.1.6 or greater to be absolutely sure this patch will work for you. If you have different version, see the following URLs to understand whether you need it at all, and if it should be applied in different way:
http://dev.mysql.com/doc/refman/5.1/en/ ... tions.html
http://dev.mysql.com/doc/refman/5.0/en/ ... tions.html
##############################################################################
*** zm_db.cpp.orig 2009-11-25 15:02:12.000000000 -0600
--- zm_db.cpp 2009-11-25 12:17:43.000000000 -0600
***************
*** 32,37 ****
--- 32,38 ----
Error( "Can't initialise structure: %s", mysql_error( &dbconn ) );
exit( mysql_errno( &dbconn ) );
}
+ my_bool dbreconnect = 1; mysql_options(&dbconn, MYSQL_OPT_RECONNECT, &dbreconnect);
std::string::size_type colonIndex = staticConfig.DB_HOST.find( ":/" );
if ( colonIndex != std::string::npos )
{
##############################################################################
--- zma
The program coredumps in the code related to linked monitors. I figured out I accidentally configured linked monitor for the one in troubles. When I removed it, everything went smoothly. I am leaving the rest to the developers.
--- zmc
The following patch fixed segmentation fault, but it is not the fix for the problem itself.
I believe when the camera returns zero length response, zma thinks the communication is lost, disconnects the file descriptor, and returns the code 0:
int RemoteCameraHttp::ReadData( Buffer &buffer, int bytes_expected )
{
....
Debug( 3, "Socket closed" );
Disconnect();
return( 0 );
....
}
I suspect this return code is not handled properly, and another attempt to read is performed on closed descriptor, which causes coredump. The patch allows to avoid it and finish the program properly instead. It is a hack, and I am leaving the rest to the developers.
##############################################################################
*** zm_remote_camera_http.cpp.orig 2009-11-26 14:30:38.000000000 -0600
--- zm_remote_camera_http.cpp 2009-11-26 14:12:00.000000000 -0600
***************
*** 129,134 ****
--- 129,140 ----
int RemoteCameraHttp::ReadData( Buffer &buffer, int bytes_expected )
{
+ if(sd < 0)
+ {
+ Error( "Bad descriptor" );
+ return -1;
+ }
+
fd_set rfds;
FD_ZERO(&rfds);
FD_SET(sd, &rfds);
##############################################################################
--- MySQL
I improved package stability with the following patch. Sometimes, when there is not enough activity, the connection to MySQL times out, and the program needs to restart. In case of zma it may cause the lost of an event, which occured after long period of inactivity. This patch allows automatic reconnection to MySQL. You must run MySQL version 5.1.6 or greater to be absolutely sure this patch will work for you. If you have different version, see the following URLs to understand whether you need it at all, and if it should be applied in different way:
http://dev.mysql.com/doc/refman/5.1/en/ ... tions.html
http://dev.mysql.com/doc/refman/5.0/en/ ... tions.html
##############################################################################
*** zm_db.cpp.orig 2009-11-25 15:02:12.000000000 -0600
--- zm_db.cpp 2009-11-25 12:17:43.000000000 -0600
***************
*** 32,37 ****
--- 32,38 ----
Error( "Can't initialise structure: %s", mysql_error( &dbconn ) );
exit( mysql_errno( &dbconn ) );
}
+ my_bool dbreconnect = 1; mysql_options(&dbconn, MYSQL_OPT_RECONNECT, &dbreconnect);
std::string::size_type colonIndex = staticConfig.DB_HOST.find( ":/" );
if ( colonIndex != std::string::npos )
{
##############################################################################
-
bristoldave
- Posts: 48
- Joined: Thu Mar 05, 2009 9:42 pm
-
bristoldave
- Posts: 48
- Joined: Thu Mar 05, 2009 9:42 pm
Just as an update to this, pskovalko's fix doesn't appear to have worked on my system, I'm still getting;
So it's erroring as per the patch above ("Bad Descriptor") but is still crashing.
Does anyone have any ideas?
Code: Select all
12/17/09 11:11:08.321913 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/178 [Expecting 1448 bytes]
12/17/09 11:11:08.321934 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/203 [Read 1448 bytes]
12/17/09 11:11:08.322370 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/178 [Expecting 1448 bytes]
12/17/09 11:11:08.322391 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/203 [Read 1448 bytes]
12/17/09 11:11:08.322764 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/178 [Expecting 600 bytes]
12/17/09 11:11:08.322786 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/203 [Read 600 bytes]
12/17/09 11:11:08.322801 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/944 [Got end of image by length, content-length = 48973]
12/17/09 11:11:08.322846 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/1011 [Returning 48973 (48973) bytes of captured content]
12/17/09 11:11:08.768996 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/106 [Connected to host, socket = 4]
12/17/09 11:11:08.769062 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/127 [Request sent]
12/17/09 11:11:08.778959 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/173 [Socket closed]
12/17/09 11:11:08.779004 zmc_m1[3414].DB3-zm_remote_camera_http.cpp/782 [Unable to extract entire header from stream, continuing]
12/17/09 11:11:08.779021 zmc_m1[3414].ERR-zm_remote_camera_http.cpp/135 [Bad descriptor]
12/17/09 11:11:08.779198 zmc_m1[3414].ERR-zm_remote_camera_http.cpp/584 [Unable to read header]
12/17/09 11:11:08.779281 zmc_m1[3414].ERR-zm_remote_camera_http.cpp/1050 [Unable to get response]
12/17/09 11:11:08.779357 zmc_m1[3414].ERR-zmc.cpp/260 [Failed to capture image from monitor 1 (0/1)]
Does anyone have any ideas?
-
bristoldave
- Posts: 48
- Joined: Thu Mar 05, 2009 9:42 pm
Yeah fair enough 
I guess zoneminder needs to test if a stream is still open before it reads to it.
Better yet, change the check where it tests whether the latest image is 0 bytes and if it is, instead of disconnecting the stream, reload the image. Can't be hard but I'm struggling to work out how to do this.
I guess zoneminder needs to test if a stream is still open before it reads to it.
Better yet, change the check where it tests whether the latest image is 0 bytes and if it is, instead of disconnecting the stream, reload the image. Can't be hard but I'm struggling to work out how to do this.