Apache NOT responding

[henriquejf]

Rony000, if i understand you correctly, the implementation of

Code: Select all

if ( $numSockets === false || $numSockets < 1 ) 

on stream.php would then solve the problem handling the zero as number of sockets, but it doesn´t seem to fix the problem tough (for me);

Do you think you had a correction for the problem in stream.php ? did you manage to get it running as it should ?


I still have to restart apache to get things running correctly time to time; no log entry helps me finding the solution until this moment;

sometimes, my error.log warns about an ajax error related to tmp/zms-908776s.sock and to /var/www/ajax/stream.php on line about 51
but not all the times that the problem happens, so i cant relate it for sure;

tks for any help !

[henriquejf]

In an effort to solve, or even organize related problems, i am asking users experiencing **APACHE HANGS** to double check what has been tried before by other users, so we don´t loose our time and get things working as expected;

I am not even sure that it is a ZM bug, but i am sure that many people have to restart apache from time to time, just as me; One user found it useful to restart apache as a cron job every x minutes, another user proposed to use http://jzmconsole.securitykit.net/ as a workaround for the problem when it happens in the montage view, but i am sure that we´ll have that fixed soon by the community, so we don´t have to give up and try the mentioned workarounds;


apache hanging problems - related, duplicated (or not) topics
==============================================================


** Maxopenfile reached & machine hangs zm problem ?
http://www.zoneminder.com/forums/viewtopic.php?t=13486

** apache not responding
http://www.zoneminder.com/forums/viewtopic.php?t=13322

** Apache instances/responsiveness issue...solved(?)
http://www.zoneminder.com/forums/viewto ... c&start=15

** Zoneminder UNSTABLE & HANGS, needs frequent restarting
http://www.zoneminder.com/forums/viewto ... 1196#51196

** hundreds of httpd processes
http://www.zoneminder.com/forums/viewtopic.php?t=13588


If some experienced programmer could give us a light, it would be so kind !!

[cordel]

See my other post to your question.

[henriquejf]

I have a simple suggestion of a workaround that works for me: close all instances of your browser and open it again;

Altough it seems to work, you may notice acumulated sock files under /tmp that may indicate that this is a real problem;

I have my experiences and logs documented in
http://www.zoneminder.com/forums/viewtopic.php?p=52491

hope it helps someone;

I still experience these "hangs" (that in fact aren´t apache hanging as you restart your browser and have things working again) with zm 1.24.1 in the latest SVN 2862 (as of may 6th, 2009)

[matias.kippes]

Hello everyone, I'm having the same problem. Probe amend stream.php without positive results.
Also edit the Apache configuration
MaxKeepAliveRequests 100 -----> 500
KeepAliveRequests 15 ---------> 5
This is merely drives the apache connections reach the top faster.

Someone comes up with something else?
Thanks

[littlej]

i'm having a strange problem with zm, afther few tries and recompiles of zm 1.24.2 my apache hangs when i try to vie events.
it displays one frame from it and hangs, if i restart works again till i try to view an event.
i have no idea how to debug this, since other pages i have hosted on same box works ok so apache is working but not working with zoneminder or something like that, and zoneminder its also working recording events and all stuff but i canot refresh the page untill i restart apache.
i have enabled full debug on zoneminder, but nothing apears on logs, i added log level debug into apache conf but still i dont see any errors, every log i check it looks ok... i have no idea where to start this, can anyone help?

ps first time i have builded zm worked ok, but now it doesnt anymore, and i dont wana do a fresh instal

thanks

[chr]

show ulimit -a . Default Value is 1024 for open Files. Apache needs more.
I use ulimit -n 15000. Now it works fine.

Maybe it helps you

[Abner]

This was happening to me as well. I went and looked at opened file handles and noticed:

[root@localhost httpd]# lsof | grep php | grep session
httpd 12791 apache 17uW REG 9,1 458 134021446 /var/lib/php/session/sess_tou1loomlmvkae4d154psuie73
httpd 12793 apache 17u REG 9,1 458 134021446 /var/lib/php/session/sess_tou1loomlmvkae4d154psuie73
httpd 12794 apache 17u REG 9,1 458 134021446 /var/lib/php/session/sess_tou1loomlmvkae4d154psuie73
httpd 12798 apache 17u REG 9,1 458 134021446 /var/lib/php/session/sess_tou1loomlmvkae4d154psuie73

So it's the PHP Session. It makes sense that closing the browser would fix this, since it would give you a new Session ID.

I added session_write_close() right above session_start() in index.php and this seems to have fixed it! Thanks to whoever suggested that!

[MarcoP]

I'm trying to look into it my self because I'm having the same issue.

With Firefox, since it uses more server connections then other browsers, its easy to duplicate the problem.

However the following code should be required by default even if not strictly related to this issue

Code: Select all

if (ini_get('session.auto_start')) {
	session_destroy();
	session_write_close();
}

Then a quick patch would be to have a __destruct method within a session class to automatically close session as well doing other possible cleanups every time the script ends.
__destruct is available for PHP 5 users only.

If I'm not mistaken, I heard some users having issues with auth when using wrong password or username and they were not able to log in again untill the restart the browser. A session not properly closed can create this issue.

[MarcoP]

Easy zoneminder-wide solution compatible with PHP 4 >=

index.php change

Code: Select all

ini_set( "session.name", "ZMSESSID" );
session_name("ZMSESSID");

to

Code: Select all

function zm_shutdown() {
	session_write_close();
}
register_shutdown_function('zm_shutdown');

ini_set( "session.name", "ZMSESSID" );
session_name("ZMSESSID");

However my personal extended version looks like

Code: Select all

if (ini_get('session.auto_start')) {
	session_destroy();
	session_write_close();
}

function zm_shutdown() {
	session_write_close();
}
register_shutdown_function('zm_shutdown');

ini_set( "session.name", "ZMSESSID" );
session_name("ZMSESSID");

[MarcoP]

While digging ...

the previous patch does help a lot but it's not the fix, its just a partial fix.

Firefox seems to be the issue, probably it is, probably not.

A browser using cambozola doesn't seems to have this issue as much a browser that support the streaming.

Code: Select all

netstat -s|grep passive

The above will output a constant increasing value for streaming browsers but a fixed value for cambozola's enabled browsers.

So something in the streaming it's causing this.
Connections are increasing, sessions are not closed and thats when things get ugly.

The patch I've provided will close the sessions, apparently fixing the problem. But the real issue it's still there, this justify why there are so many opened file descriptors.

[MarcoP]

lol sorry guys ... just posting any finding

Applying my previous patch I was able to play around long enough.

KeepAlive Off
worse scenario but can get worse
To reproduce:
1 - start apache with KeepAlive Off
2 - open montage and be patience
3 - close montage and repeat step 2

Code: Select all

# apachectl fullstatus | grep -E 'Srv|POST'

Srv   PID     Acc    M  CPU  SS Req Conn Child Slot  Client           VHost                                   Request
0-0  24824 0/87/87   _ 48.51 3  89  0.0  0.12  0.12 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
1-0  24825 0/87/87   _ 44.26 0  118 0.0  0.18  0.18 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
4-0  24828 0/88/88   _ 49.84 0  130 0.0  0.19  0.19 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
5-0  24829 0/58/58   _ 43.79 3  80  0.0  0.12  0.12 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1

Srv   PID     Acc    M  CPU  SS Req Conn Child Slot  Client           VHost                                   Request
2-0  24826 0/62/62   _ 50.98 5  72  0.0  0.17  0.17 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
3-0  24827 0/87/87   _ 37.97 5  76  0.0  0.19  0.19 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
4-0  24828 0/102/102 _ 50.18 2  69  0.0  0.20  0.20 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
8-0  24833 0/139/139 _ 8.33  2  55  0.0  0.24  0.24 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1

KeepAlive On
MaxKeepAliveRequests 4000 (I don't use unlimited requests to avoid memory leaks)
Follow step 2 and 3

Code: Select all

# apachectl fullstatus | grep -E 'Srv|POST'

Srv   PID     Acc     M  CPU  SS  Req Conn Child Slot  Client           VHost                                   Request
0-0  27597 64/125/125 K 9.29  1   109 10.3 0.13  0.13 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
1-0  27598 2/83/83    K 4.69  1   153 0.3  0.16  0.16 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
3-0  27600 63/97/97   K 13.79 1   158 10.2 0.11  0.11 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
7-0  27604 2/3/3      W 4.36  576 0   0.3  0.00  0.00 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1

Srv   PID     Acc     M  CPU   SS  Req  Conn Child Slot  Client           VHost                                  Request
3-0  27600 58/180/180 K 16.01 0    104  9.3  0.12  0.12 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
6-0  27603 55/81/81   K 9.95  2    113  36.0 0.22  0.22 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
7-0  27604 2/3/3      W 4.36  1056 0    0.3  0.00  0.00 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
17-0 28399 15/58/58   K 2.29  2    131  2.4  0.09  0.09 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
18-0 28400 15/27/27   K 6.55  3    89   2.4  0.10  0.10 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1

First of all you will notice the high CPU usage, but whats also important is "Acc". KeepAlive On will re-use the same connection instead of creating a new one speeding up things.

Without my patch I was able to understand something, KeepAlive is still On but have a look at the following.
Apache Hangs for over 2200 seconds

Code: Select all

Srv   PID     Acc     M CPU   SS   Req Conn Child Slot  Client           VHost                                   Request
3-0  29534 21/23/23   W 1.45  2231 0   3.4  0.01  0.01 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
6-0  29537 0/8/8      W 2.17  2229 0   0.0  0.01  0.01 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
7-0  29538 0/3/3      W 1.84  2229 0   0.0  0.01  0.01 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
10-0 29551 0/20/20    W 0.56  2227 0   0.0  0.01  0.01 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1
11-0 29582 0/2/2      W 0.07  2226 0   0.0  0.01  0.01 10.1.2.3  localhost.localdomain POST /zm/index.php HTTP/1.1


lsof | grep sess
httpd     29534    apache   16uW     REG                8,1      524        111 /var/lib/php/session/sess_u318deekh3vqkfmtk02emvvui5
httpd     29537    apache   16u      REG                8,1      524        111 /var/lib/php/session/sess_u318deekh3vqkfmtk02emvvui5
httpd     29538    apache   16u      REG                8,1      524        111 /var/lib/php/session/sess_u318deekh3vqkfmtk02emvvui5
httpd     29551    apache   16u      REG                8,1      524        111 /var/lib/php/session/sess_u318deekh3vqkfmtk02emvvui5
httpd     29582    apache   16u      REG                8,1      524        111 /var/lib/php/session/sess_u318deekh3vqkfmtk02emvvui5

Apache hangs at session_start() in index.php

So at the end I have my patch in place and KeepAlive On with 4000 MaxReqs.

Hope it helps, cheers.

[jfkastner]

thanks for the investigation - i had some ideas about it too at

http://www.zoneminder.com/forums/viewtopic.php?t=15145

however i believe it's NOT FF - i've tried 3, 3.5, 3.6, 3.7 and all have the same problem (all w/o cambo)

also most of the time i go thru a squid proxy thru the WAN to my remote server, same stalls with montage with or w/o proxy (but very different connection limits etc)

my guess it's in montage my WAN isn't fast enough to display all the frames from all the cams and ZM keeps sending/creating new ones before the older ones are displayed properly (and all sockets etc cleared)

funny is the timecode seems to 'jump' sometimes - if you run a cam at 1 fps you'll see it randomly increase rapidly till realtime

[johnnytolengo]

hello, today I got a log from apache but I don't know what it means.

Code: Select all

==> /var/log/apache2/access.log <==
212.158.129.78 - - [26/May/2010:09:47:19 +0200] "HEAD / HTTP/1.0" 200 420 "-" "-"

of course apache was not reacting at the requests from http.

some idea?

[jhetrick62]

In total, I made 4 changes as I found that not any one of them did the trick by itself, but together, it seemed to run better and FireFox crashed much less often, if at all.

1. I modified ajax/stream.php
from: if ( $numSockets === false )

to: if ( $numSockets === false || $numSockets < 1 )

2. I employed MarcoP's fix above into the index.php file

3. I followed jfkastner's post: http://www.zoneminder.com/forums/viewtopic.php?t=15145
and made those changes to /etc/apache2/apache2.conf file
- As he did them in Webmin, they were defined slightly differently in the actual .conf file so you need to set to the following:
StartServers 10
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 10

4. Finally I also edited the /etc/security/limits.conf file to add these lines.
* - nofile 4096
root - nofile 4096
www-data - nofile 4096

In total, all of these together seemed to make a small impact each. In the end, it became more and more stable. All of this information was gleaned from various different posts in the Zoneminder forums, so the credit goes to all of the folks that worked this out, I just put them all together.

For the record, my system is running in Sun VirtualBox on a Centos server. It is built with the Lucid 10.04 LTS _x64 Server edition with the base install. Zoneminder and FFmpeg were both added with standard repos configs: sudo aptitude install ffmpeg zoneminder

I added Camboloza and I attached the /var/cache/zoneminder to an NFS share to enable use of a 3.6tb storage drive as this will have 20+ cameras when done.

Otherwise with the modifications above and the stock install, it just runs! I hope that this helps someone else and I surely have enjoyed over the last 2 years, the help gleaned from many other parties in these forums.

Jeff