Apache NOT responding

c0mputerking · Post by **c0mputerking** » Mon Mar 16, 2009 9:35 pm

Hello all i am starting a new post in regards to this thread because of the misleading title. Apache instances/responsiveness issue as this problem is NOT solved.

http://www.zoneminder.com/forums/viewto ... c&start=15

Also i would like to add to my posts mentioned in the thread above. Today i have experienced almost complete Apache lockup. Seems to be triggered while using montage view one camera out of 5 will freeze (ie timestamp does not change) and give some ridiculously high fps (199.0 fps sometimes 1999.0 fps on the ip camera). When this happens i cannot get apachectl status to respond. However the zm logs look ok still capturing frames sending emails and the like. Apache is giving this

[warn] [client 192.168.1.1] Timeout waiting for output from CGI script /var/lib/zm/cgi-bin/nph-zms, referer: http://computerking.surveillanceking.ne ... ge&group=0
[Mon Mar 16 15:08:45 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting. I have MaxKeepAliveRequests in hopes of fixing the error I will let you know if this works soon testing right now.

#MaxKeepAliveRequests 100
MaxKeepAliveRequests 500

When i restart Apache it takes a very long time like 1min to shutdown and the log is full of these

Code: Select all


[Mon Mar 16 14:55:39 2009] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8j DAV/2 configured -- resuming normal operations
[Mon Mar 16 15:05:21 2009] [warn] [client 192.168.1.1] Timeout waiting for output from CGI script /var/lib/zm/cgi-bin/nph-zms, referer: http://computerking.surveillanceking.net/?view=montage&group=0
[Mon Mar 16 15:08:45 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Mar 16 15:24:57 2009] [warn] child process 13647 still did not exit, sending a SIGTERM
[Mon Mar 16 15:24:57 2009] [warn] child process 13648 still did not exit, sending a SIGTERM

....SNIP ALOT OF THESE....

[Mon Mar 16 15:25:01 2009] [warn] child process 13752 still did not exit, sending a SIGTERM
[Mon Mar 16 15:25:01 2009] [warn] child process 13753 still did not exit, sending a SIGTER

newvisionantenna · Post by **newvisionantenna** » Mon Mar 16, 2009 9:56 pm

What was the error again your getting?

j/k wish i could help but haven't seen that problem yet

c0mputerking · Post by **c0mputerking** » Mon Mar 16, 2009 10:10 pm

mostly these errors like about 100 of them

Code: Select all

[Mon Mar 16 15:25:47 2009] [error] child process 13989 still did not exit, sending a SIGKILL
[Mon Mar 16 15:25:48 2009] [error] could not make child process 13746 exit, attempting to continue anyway

and one of these

Code: Select all

[warn] [client 192.168.1.1] Timeout waiting for output from CGI script /var/lib/zm/cgi-bin/nph-zms, referer: http://computerking.surveillanceking.net/?view=watch&mid=2

also one of these

Mon Mar 16 15:08:45 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting

however i have increased my MaxKeepAliveRequests from 100 to 500 and things seem to be working better but i am still testing ie waiting for lockup however so far so good here is output of apachectl status has been over 1/2 one hour now and only one timeout waiting for CGI script error so far.

pache Server Status for localhost

Server Version: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8j
DAV/2

Server Built: Jan 29 2009 15:13:41
__________________________________________________________________

Current Time: Monday, 16-Mar-2009 16:08:14 MDT
Restart Time: Monday, 16-Mar-2009 15:32:01 MDT
Parent Server Generation: 0
Server uptime: 36 minutes 13 seconds
Total accesses: 360 - Total Traffic: 468 kB
CPU Usage: u8.04 s2.92 cu129 cs0 - 6.44% CPU load
.166 requests/sec - 220 B/second - 1331 B/request
9 requests currently being processed, 6 idle workers

_WWW__WW_WWWW__.................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

ladislav2 · Post by **ladislav2** » Thu Mar 19, 2009 12:27 pm

Similar problem here. Zoneminder keeps running, but apache has to be always restarted, after bunch of request it is locked up again.

lonewolf · Post by **lonewolf** » Fri Mar 20, 2009 12:06 am

Same issue here. I switched from nph-zms to zms to no avail. It's gotten so bad I've written a cron script that monitors Apache and restarts it when it locks. I have user authentication turned on in ZM if that makes a difference. KeepAlive is off.

Not sure if it's related, but I'm also seeing these errors in Apache's error log:

Code: Select all

[Thu Mar 19 20:07:18 2009] [error] [client 172.19.9.102] socket_sendto( /tmp/zms-410256s.sock ) failed: Connection refused, referer: http://172.19.8.1/zm/index.php?view=montage&group=0

[Thu Mar 19 20:07:18 2009] [error] [client 172.19.9.102] array (\n  0 => \n  array (\n    'file' => '/usr/share/zoneminder24/www/ajax/stream.php',\n    'line' => 51,\n    'function' => 'ajaxError',\n    'args' => \n    array (\n      0 => 'socket_sendto( /tmp/zms-410256s.sock ) failed: Connection refused',\n    ),\n  ),\n  1 => \n  array (\n    'file' => '/usr/share/zoneminder24/www/index.php',\n    'line' => 116,\n    'args' => \n    array (\n      0 => '/usr/share/zoneminder24/www/ajax/stream.php',\n    ),\n    'function' => 'require_once',\n  ),\n), referer: http://172.19.8.1/zm/index.php?view=montage&group=0

[Thu Mar 19 20:07:18 2009] [error] [client 172.19.9.102] socket_sendto( /tmp/zms-943755s.sock ) failed: No such file or directory, referer: http://172.19.8.1/zm/index.php?view=montage&group=0

[Thu Mar 19 20:07:18 2009] [error] [client 172.19.9.102] array (\n  0 => \n  array (\n    'file' => '/usr/share/zoneminder24/www/ajax/stream.php',\n    'line' => 51,\n    'function' => 'ajaxError',\n    'args' => \n    array (\n      0 => 'socket_sendto( /tmp/zms-943755s.sock ) failed: No such file or directory',\n    ),\n  ),\n  1 => \n  array (\n    'file' => '/usr/share/zoneminder24/www/index.php',\n    'line' => 116,\n    'args' => \n    array (\n      0 => '/usr/share/zoneminder24/www/ajax/stream.php',\n    ),\n    'function' => 'require_once',\n  ),\n), referer: http://172.19.8.1/zm/index.php?view=montage&group=0

/LW

chandi · Post by **chandi** » Fri Mar 20, 2009 1:33 am

I have the exact same problem. When I click on my only monitor I see a live stream. If I click on an event it shows the replay and then if I click on either stills/frames/videos apache stops respondind and I get the same log as you do.

I have increased the max keep alive requests, and similar settings in firefox to no avail.

I have upgraded apache and it still hangs.

That's getting very frustrating.

Chandi

lonewolf · Post by **lonewolf** » Fri Mar 20, 2009 1:48 am

Fount it!

To reproduce: Open the montage window. After it loads, right-click the page and select 'Stop' (firefox) or otherwise cause the window to close uncleanly. ZM no longer responds to that session.

Problem: PHP is hanging on the session_start() call. You can verify this by causing the session to hang as per above and edit index.php and put an 'exit;' right above session_start(); the index page now loads but it's blank. Now move the 'exit;' down to just after session_start(); trying to view the index page will now hang indefinitely. As the PHP script never exits, Apache will eventually run out of children and stop taking requests.

PHP bug 22526 says to use session_write_close() on all the pages that uses sessions to fix this. I'm gonna try it and see if it works here.

/LW

chandi · Post by **chandi** » Fri Mar 20, 2009 3:46 am

Ok great. But why does it seem to affect only 1.24.xx and not the earlier versions ?

And why doesn't it affect everybody ?

Thanks,

Chandi

c0mputerking · Post by **c0mputerking** » Sat Mar 21, 2009 12:01 am

I am guessing it has something to do with the rewritten web interface, but like i say that is just a guess. However you question about only some of being affected by this problem seems valid. I am running the version off the ZMLarch live CD produced by rmedlin so i am not exactly sure how he compiled ZM maybe it has to do with some compile options.

I really hope we can get this problem sorted soon I have wasted alot of time tiring to make it work on my own. A first i thought maybe it was that my hardware could not keep up so I move ZM to a quad core and it still broke after a couple of hours

In this condition ZM 1.24.1 is unusable IMO I an quite glad that there are others having the same issue better chance of getting solved sooner.

newvisionantenna · Post by **newvisionantenna** » Sat Mar 21, 2009 3:26 pm

The user name and password is admin/admin.

Let me know if you see the errors you mention. Yes, it's true I ask for a donation or activation but I've agreed to return a portion of the money to the developer of Zoneminder. I haven't given anything yet because only a handful of people have bought a key, the rest I've given out a free key for them to use. I have some interesting things in the works and finally created the ultimate UVC/MjpgStreamer platform with wireless and money making hotspot ability. It took a lot of work but I'll make a whole post about it and how it works nicely with Zoneminder.

lonewolf · Post by **lonewolf** » Tue Mar 24, 2009 12:29 am

I compiled with:
make clean && ./configure --prefix=/opt/zoneminder24 --with-libarch=lib64 --with-webdir=/usr/share/zoneminder24/www --with-cgidir=/usr/share/zoneminder24/cgi-bin --with-ffmpeg=/opt && make

Adding a bunch of session_write_close() (and calling session_start() again where needed) helped a LOT. Each time the error occurs a single Apache child hangs, but it does not lock you out anymore. The max children setting is high enough to not hit the limit before Apache is restarted (when the logs are rotated each night).

I believe the hang is AJAX related as the hanging page is a POST request to index.php. I'll do some packet capturing later tonight to see what exactly it is asking for.

/LW

c0mputerking · Post by **c0mputerking** » Tue Mar 24, 2009 12:52 am

Thanks Lone Wolf not sure about the compile bit as i am using ZMLarch CD, but i am stuck on this problem too let me know what i can do to help.

lonewolf · Post by **lonewolf** » Tue Mar 24, 2009 3:21 am

Ok, here's the deal:

There is a race condition in ajax/stream.php in which if the connection is closed after the socket_sendto() call but before the response is received (sometime during the socket_select() call), socket_select() returns 0 (not FALSE) which is not handled and so it allows socket_recvfrom() to be called. This then blocks indefinitely while waiting for a response which is never going to come as the other end of the socket is already closed.

To fix: on or about line 60 in ajax/stream.php change

Code: Select all

if ( $numSockets === false )

to

Code: Select all

if ( $numSockets === false || $numSockets < 1 )

/LW

c0mputerking · Post by **c0mputerking** » Tue Mar 24, 2009 5:45 am

WOW Lonewolf you are a fellow genius heheh, made the change you suggest and It seems to be working very well now.

Have hammered on it for awhile now and no lockups this makes me unimaginably happy thanks for your hard work, and your posts i will keep you posted mostly if things go wrong heehee.

I guess we will see if it is stable first overnight, and then a couple of days. Then hopefully FreeBSD stable think my record for reboot was about 4 years however not doing anything as intense as video.

Normando · Post by **Normando** » Tue Mar 24, 2009 6:07 am

lonewolf wrote:Ok, here's the deal:

There is a race condition in ajax/stream.php in which if the connection is closed after the socket_sendto() call but before the response is received (sometime during the socket_select() call), socket_select() returns 0 (not FALSE) which is not handled and so it allows socket_recvfrom() to be called. This then blocks indefinitely while waiting for a response which is never going to come as the other end of the socket is already closed.

To fix: on or about line 60 in ajax/stream.php change
Code: Select all
if ( $numSockets === false )
to
Code: Select all
if ( $numSockets === false || $numSockets < 1 )
/LW

Brilliant! I will test and report back the results.