ZoneMinder Forums

At least I've hammered on it for >5mins. playing event after event, etc. and there's no hanging from apache side.

I found out using apache status mod (http://myipaddr/server-status) that worker processes were being left around due to long keepalive.

By making the following change I now have no known apache problems...5 may not be ideal setting but I know what to play with now.

Nothing on forums references this setting...amazing I'm the only one who stumbled on this.

#
# KeepAliveTimeout: Number of seconds to wait for the next request
# from the same client on the same connection.
#
#KeepAliveTimeout 15
KeepAliveTimeout 5

The above file was for me (ubuntu 8.10) in /etc/apache2/apache2.conf.

Thanks for this. Though I'm not sure why this would slow things down, surely having a process waiting around to service a request would speed things up rather than having to start a new child? I'm not an apache expert though.

Processes weren't waiting...that's the problem. They were in 'W' state which means writing to client connection...which has been closed but takes a long time to register.

Each open of video in browser from apachehost/zm results in a single apache thread going into W state and not returning quickly.

So viewing just a small handful of streams results in 10s of processes in W state.

I could never hammer video play/etc. until I lowered this setting...now I can hammer video and apache never hangs.

Never hangs...that is until I went back up to 7 cameras @ 640x480...but this was problem...not apache I think.

Below is text from server-status page:

Apache Server Status for 192.168.1.10

Server Version: Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.1 with Suhosin-Patch
Server Built: Sep 19 2008 13:44:17

Current Time: Friday, 20-Feb-2009 00:32:48 PST
Restart Time: Thursday, 19-Feb-2009 23:37:54 PST
Parent Server Generation: 0
Server uptime: 54 minutes 53 seconds
10 requests currently being processed, 5 idle workers

_WWW_WWWWW_WW__.................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

PID Key:

21693 in state: _ , 16583 in state: W , 16584 in state: W
16585 in state: W , 21694 in state: _ , 16595 in state: W
17962 in state: W , 18116 in state: W , 18120 in state: W
18127 in state: W , 21695 in state: _ , 20849 in state: W
20854 in state: W , 21696 in state: _ , 21697 in state: _

From above it is '10 requests currently being processed' which is problem...no they aren't conn was closed earlier.

KeepAliveTimeout 5 is the default on a Slack system

I'm running ubuntu 8.10.

Interesting it's set by default to 3x slackware setting.

I am running ZM 1.24 on a Slackware 12.2 install and I have had the same issue with apache intermittently locking up.

When running two cam's on a BTTV 878 card it happens once or twice a week.

I added a USB_CAM today and it now happens every hour.

Just tried your KeepAliveTimeout suggestion to see if that clears things up.

BTW on my stock install of Slackware 12.2 my config fill didn't have a KeepALiveTimeout option set at all.

Cheers

No luck, the KeepAliveTimeout "fix" didn't help.

I am using Firefox which may be part of the problem as many people have similar issues with version 3.

What does apache-status module say?

What does "ps aux | grep apache" say?

apache shows that it is running, however it does not respond to requests.

ps aux | grep apache shows the following repeating over and over.


apache 6640 0.0 0.4 60760 9188 ? S 22:10 0:00 /usr/sbin/httpd -k start
apache 6641 0.0 0.4 60760 9136 ? S 22:10 0:00 /usr/sbin/httpd -k start
apache 6642 0.0 0.3 60208 7652 ? S 22:10 0:00 /usr/sbin/httpd -k start
apache 6644 0.0 0.3 60208 7652 ? S 22:10 0:00 /usr/sbin/httpd -k start
apache 6649 0.0 0.3 60208 7652 ? S 22:10 0:00 /usr/sbin/httpd -k start
apache 6650 0.0 0.3 60208 7652 ? S 22:10 0:00 /usr/sbin/httpd -k start

That is the clue to part of the problem. Since apache isn't actually stopped the "httpd -k start" will only respond that httpd is already running.

To get around this you need to call "httpd -k restart" which will get apache running regardless of the current state of httpd.

I am assuming that one of the Zoneminder perl scripts is sending the start command and an quick edit could solve the symptom.

It still doesn't answer the root cause, but in my case it linked to the USB_CAM that I added today and possible Firefox.

I will check more of the logs over the weekend to try and pin down my issue.

Cheers

enable server-status module and track apache's performance/stats with it.

You can also turn "ExtendedStatus On" in apache conf.

http://httpd.apache.org/docs/1.3/mod/mod_status.html

It isn't zm creating apache instances...apache is doing this.

Did you restart apache after changing keepalive?

phowarth wrote:BTW on my stock install of Slackware 12.2 my config fill didn't have a KeepALiveTimeout option set at all.

If you uncomment the default settings Include statement on line 467 (or thereabouts) in /etc/httpd.conf apache will use the Slack default of 5. Otherwise, apache will use the "apache2" default of 15.

mklein wrote:enable server-status module and track apache's performance/stats with it.

You can also turn "ExtendedStatus On" in apache conf.

That can be found in /etc/httpd/extra/httpd-info.conf and is "off" by default.

Uncomment the Include /etc/httpd/extra/httpd-info.conf statement on line 448 to use this file.

Thanks for the tips on apache. I have used httpd.conf in the past but didn't notice that you could call other configuration files from within.

I have updated the recommended files and will report back with any additional outputs.

I did delete my USB Cam last night and ZM/Httpd has run for 24 hours with no issues.

Now that I have made the changes to apache config's I will put the USB CAM back in to see what happens.

Paul

Alrighty... I got apache server-status up and running, thanks for the tips. I'll add one more.. make sure that you add your local subnet to the following in /etc/httpd/extra/httpd-info.conf or you will get permission denied when you try to load server-status.

<Location>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 10.0.10. <------- your subnet or specific ip addy here.
</Location>

OK I re installed my USB CAM and this time it lasted about an hour before httpd froze.

server-status was unresponsive, so I restarted and did a quick refresh and here is the output.

CPU Load is 131% - Ouch!

Also the number of open connections (if am reading this right) is huge on the machine with a montage view open on firefox.

Current Time: Saturday, 21-Feb-2009 21:40:03 EST
Restart Time: Saturday, 21-Feb-2009 21:40:00 EST
Parent Server Generation: 7
Server uptime: 3 seconds
Total accesses: 101 - Total Traffic: 76 kB
CPU Usage: u2.62 s.2 cu1.11 cs0 - 131% CPU load
33.7 requests/sec - 25.3 kB/second - 770 B/request
12 requests currently being processed, 0 idle workers

WWWWWWWWWWWW....................................................
................................................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-7 22382 0/18/18 W 0.50 0 0 0.0 0.00 0.00 10.0.10.200 10.0.10.200 GET /index.php HTTP/1.1
1-7 22384 0/19/19 W 1.17 0 0 0.0 0.02 0.02 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
2-7 22385 0/12/12 W 0.29 0 0 0.0 0.01 0.01 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
3-7 22386 0/15/15 W 1.02 0 0 0.0 0.02 0.02 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
4-7 22387 0/19/19 W 0.56 0 0 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
5-7 22392 0/5/5 W 0.14 0 0 0.0 0.01 0.01 10.0.10.200 10.0.10.200 GET /server-status HTTP/1.1
6-7 22406 0/4/4 W 0.13 0 0 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
7-7 22408 0/7/7 W 0.20 0 0 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
8-7 22417 0/1/1 W 0.02 0 0 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
9-7 22419 0/0/0 W 0.00 0 1680178014 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
10-7 22420 0/1/1 W 0.00 0 0 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1
11-7 22421 0/0/0 W 0.00 0 1680177982 0.0 0.00 0.00 10.0.10.126 10.0.10.200 POST /index.php HTTP/1.1

Phowarth - Just out of curiousity, what is the cpu load on your system with apache sitting idle (ie, no active connections)?

Better yet, what does "top -u apache" have to say about what is going on with cpu load?