Hundreds of httpd processes and no response

[marloweos]

Hello,

ZoneMinder 1.28.1 on CentOS 7 - Linux ### 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Installed using ZmRepo.
Running as a Virtual Machine on KVM / libvirt / CentOS 6.
All IP Cameras.

Everything works fine, for a minute or two, and then Apache stops responding. On closer inspection there are hundreds of httpd processes. So I restart httpd (takes a good 30 seconds) and all is well for another minute or two. I can see the processes slowly increasing up to around 200 or more.

Using Worker.c I have limited the processes (MaxClients) to around 40. But now Apache stops responding even sooner, as soon as the processes reach 40.

So I have concluded something is not clearing up the used processed, and locking up httpd processes, which MaxClients counts, and prevents me having more.

My error log says, repeatedly (every 10 seconds):
[Sun Jul 05 22:42:57.872802 2015] [:error] [pid 9579] [client 192.168.0.2:44330] ERR [socket_sendto( /var/lib/zoneminder/sock/zms-419223s.sock ) failed: No such file or directory], referer: http://192.168.0.4/zm/index.php?view=watch&mid=16

I do have working image streams - and a lot of the responses to that error are when there are no images streams at all.

Any suggestions would be helpful.

Thank you.

[knight-of-ni]

I do have working image streams - and a lot of the responses to that error are when there are no images streams at all.

I'm not sure I understand what you are saying. Your streams are working except when they aren't???

Zoneminder can slow things down to a near halt if you have (ip) cameras that drop offline. We are aware of that and are working on it. Suggest you temporarily disable all your cameras (set function to none). Verify Apache remains responsive and your cpu does not consistently exceed ~75%.

Now, slowly enable your cameras and watch what happens. If you discover a couple of cameras are causing the slow down, then we can troubleshoot that.

[marloweos]

I found the solution, for anyone who may experience the same symptoms.

I had initially set a high frame rate on the cameras, and with 720p images in JPEG streaming, this was too demanding for the network, so I lowered the rate.

What I had forgotten, I had left another PC running at the initial setting of full FPS, which was constantly requesting more images and refreshes of the console than it could actually acknowledge and handle - I was effectively performing a DOS attack on my own set up. This is what lead Apache to spawn up a huge number of processes very quickly. Closing this browser fixed the issue instantly.

So basically, I made a silly mistake.

Thank you for your help and reply.

[knight-of-ni]

Excellent, glad you got it working.