Maxopenfile reached & machine hangs zm problem ?

[mitch]

this is only worth trying if you are already compiling from source but in short:

Code: Select all

svn co http://svn.zoneminder.com/svn/zm/trunk zoneminder
cd zoneminder
wget http://mitchcapper.com/zm_streams.patch or http://mitchcapper.com/zm_web_fixes.patch
patch -p0 < zm_streams.patch or zm_web_fixes.patch
./configure (with your configure options or see manual)
make
make install

and you should be good to go. Again if you arn't used to running from source or not from trunk wait for others to give it a shot.
Web fixes include the streams one but the streams one is the only one that could effect this.

[rdmelin]

Thanks mitch. Maybe you could put these in the patches section in the wiki. More folks are likely to find them. Or if you like I'll put them up.

[henriquejf]

Hi Everyone from the community, and tks for your time and the efforts so far !!!

This should be patched already in SVN, are you running the latest version?

i still experience problems on zm 1.24.1 in the latest SVN 2862 (as of may 6th, 2009)



summary/workaround: (when apache "seems" to be hung/ locked)
Simply close *all instances* of your browser and have a new try !!



long,long version:...
I tought i was just going to documment with logs the situation I had (apache hanging), but when i was almost ending my report, i have found a
hopefully useful info:
* Im my case, apache was not really hanging; if i had firefox not responding, i tought it was hung because only restarting apache i could manage
to have it working back; but then i tryed to access zm from the internet explorer (argh!), and it then happened to work !!! so it was not apache
hanging in the end, because IE could still access the zm ! no need to restart apache for this; on the other hand, FF still worked with any
otherwebsite, except from my zm install; i have even been able to watch an youtube video from ff during the so called "hung state" of my server,
but not my zm site still; I tried to put ff offline/online, but no changes; finally, i have found out that simply closing all instances of FF and
then opening it again solved what appeared to be a big problem;
Yes, i totally agree, it´s *stupid* how I have not tried to simply close FF and open it again, as i was so focused on restarting apache and watch
the logs to find out what was the big deal; for me, it´s now a level 1 problem, not a level 10 anymore; it may even be a bug, and *IF* it is, i
think some good soul may find out it was a session corruption/ misidentification problem, or a simple problem, because all audits and processes
reamain cool; what a shame for me !!!
Below i´ll let exposed my experiences during the so-called "apache hung state", in the hope that it will be useful for someone to
trace/investigate the situation, wich for me is no longer a big concern; actually, i now think it may be a browser´s deficciency or a default
config/parameter that doesn´t fit my exact need to have multiple streams into a browser window; i have already tried to change strem parameters of
firefox as suggested on faq, but altough they gave me better performance, they got my firefox more unstable and more prone to crash;

---xxx---

I have just upgraded my zm 1.24.1 to the latest SVN 2862 (may 6th, 2009); Unfortunately I still have got the "APACHE HANGING" simptoms; ill try to
expose some info that i hope may help to identify or reproduce the problem: (actually i think this new svn minimizes the problem, i cant measure
how exactly, but I think there are people very near to having it fixed; once again i must to tell i am not an expert and cant blame zm to have a
bug for sure)


RIG:
=====
* using a core2quad q6600 cpu, intel dg31 mobo (100% intel chipset), 2Gb ram, stable as a rock;
* using a bt878 capture card with 4 inputs (4 analog cheap cameras) that is 100% compatible and stable with the system for weeks online;

* note: When i say "apache hanging", i mean exactly that; i can still access any other services on the machine, like SSH, Webmin, mail server,
etc... When the apache is on the "hang" state, memory and cpu still remains cool, no overload noticed, nothing unusual; just a "service apache2
restart" and i have zm working back again;

* Firefox 3.0.10 with 100% original settings; (uninstall > remove all dirs > reinstall to guarantee no misconfigs)

DISTRO AND ZM IMPLEMENTATION:
==============================
* Followed excelent wiki guide for ubuntu 8.10 32bit server available below:
http://www.zoneminder.com/wiki/index.ph ... ozola-0.7)

* zm 1.24.1 in the latest SVN 2862 (as of may 6th, 2009)

REPRODUCTION of the problem: (always on wan, rare on lan, both IE and FF tested)
=================================================================================

* This problem is "most" noticiable on external access (WAN); I can still reproduce the error on the local zm machine or on another lan-side
machine, but it´s very difficult to; let me explain: when i´m accessing zm from wan, it´s a matter of opening the montage view, closing it,
opening it again from twice to 3 times, and that´s it: from this point on, no more responses from apache until the manual restarting; (no errors
on screen or on apache logs also, it just goes into an infinite loading state till timeout...)

(i have posted somewhere else a suggestion that a "close" button that could "elegantly" close the open sockets could help, maybe, as the actual
"close" on the montage view/etc just triggers a javascript closeWindow() event and counts on zms to stop automatically as there is "no one else"
on the other side of the stream )
* On the other hand, when i´m on Lan side (or on the zm localhost), it´s very difficult (but possible) to reproduce the error; actually, i have to
try hard for more than 5 minutes to open the montage views, closing them, opening a random monitor, closing it, opening the timeline, closing
it.... and then repeat this insane ritual until i get apache to hang; (as you have noticed, it´s a very forced situation, so actually i am not
giving a damm about the lan-side problem as it doesn´t bother me normally, it´s just a statement to mention; the real problem is on the "wan"
side, when the problem occurs easily as mentioned)

RELATED LOGS AND BEHAVIORS: (zm logs set to extra, level 9, restarted to apply)
==================================================================================

During normal system operation: no syslog relevant info, no apache errors, no zm_debug.log(x) errors;

### ### ### apache processes and users under zm stopped state:
: top -u www-data

top - 10:39:30 up 2:25, 1 user, load average: 0.03, 0.03, 0.00
Tasks: 135 total, 1 running, 134 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2069868k total, 632392k used, 1437476k free, 62040k buffers
Swap: 1646620k total, 0k used, 1646620k free, 422716k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14997 www-data 20 0 30316 7192 3412 S 0 0.3 0:00.09 apache2
14999 www-data 20 0 29820 4296 1136 S 0 0.2 0:00.00 apache2
15000 www-data 20 0 30272 6836 3088 S 0 0.3 0:00.04 apache2
15001 www-data 20 0 29820 4304 1136 S 0 0.2 0:00.00 apache2
15013 www-data 20 0 30376 6852 3144 S 0 0.3 0:00.06 apache2
15014 www-data 20 0 30244 6840 3140 S 0 0.3 0:00.03 apache2
15015 www-data 20 0 30376 6848 3144 S 0 0.3 0:00.06 apache2
15016 www-data 20 0 30244 6844 3144 S 0 0.3 0:00.03 apache2
15662 www-data 20 0 30128 7060 3356 S 0 0.3 0:00.13 apache2
15663 www-data 20 0 29820 4200 1100 S 0 0.2 0:00.00 apache2


### ### ### apache processes and users under zm running and with no views/clients/montages
: top -u www-data

top - 10:45:57 up 2:32, 1 user, load average: 0.21, 0.06, 0.02
Tasks: 145 total, 2 running, 143 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.9%us, 0.0%sy, 0.0%ni, 99.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2069868k total, 749436k used, 1320432k free, 62420k buffers
Swap: 1646620k total, 0k used, 1646620k free, 461528k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16356 www-data 20 0 123m 18m 9756 S 1 0.9 0:00.13 zma
16350 www-data 20 0 123m 18m 9532 S 1 0.9 0:00.12 zma
16353 www-data 20 0 123m 18m 9752 S 1 0.9 0:00.13 zma
16348 www-data 20 0 167m 50m 39m S 1 2.5 0:00.33 zmc
16359 www-data 20 0 123m 18m 9304 S 1 0.9 0:00.10 zma
14997 www-data 20 0 30188 7184 3416 S 0 0.3 0:00.22 apache2
14999 www-data 20 0 29820 4296 1136 S 0 0.2 0:00.00 apache2
15000 www-data 20 0 30160 7136 3396 S 0 0.3 0:00.09 apache2
15001 www-data 20 0 29820 4304 1136 S 0 0.2 0:00.00 apache2
15013 www-data 20 0 30376 6852 3144 S 0 0.3 0:00.06 apache2
15014 www-data 20 0 30244 6844 3144 S 0 0.3 0:00.04 apache2
15015 www-data 20 0 30376 6848 3144 S 0 0.3 0:00.06 apache2
15016 www-data 20 0 30272 6992 3264 S 0 0.3 0:00.09 apache2
15662 www-data 20 0 30128 7064 3360 S 0 0.3 0:00.13 apache2
15663 www-data 20 0 29820 4208 1104 S 0 0.2 0:00.00 apache2
16326 www-data 20 0 10668 6100 1472 S 0 0.3 0:00.02 zmdc.pl
16361 www-data 20 0 13692 10m 2764 S 0 0.5 0:00.21 zmfilter.pl
16363 www-data 20 0 11140 7812 2744 S 0 0.4 0:00.17 zmaudit.pl
16366 www-data 20 0 10540 7220 2764 S 0 0.3 0:00.15 zmwatch.pl
16369 www-data 20 0 11780 8544 2752 S 0 0.4 0:00.19 zmupdate.pl


### ### ### apache processes and users under zm running normally and with montage for 4 analog 320*240 cameras
: top -u www-data

top - 10:50:03 up 2:36, 1 user, load average: 0.12, 0.07, 0.01
Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.4%us, 3.9%sy, 0.0%ni, 88.7%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2069868k total, 892124k used, 1177744k free, 63224k buffers
Swap: 1646620k total, 0k used, 1646620k free, 553664k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16393 www-data 20 0 122m 16m 7324 S 2 0.8 0:00.22 nph-zms
16350 www-data 20 0 123m 20m 11m S 1 1.0 0:03.33 zma
16356 www-data 20 0 123m 20m 11m S 1 1.0 0:06.94 zma
16359 www-data 20 0 123m 20m 11m S 1 1.0 0:03.29 zma
16348 www-data 20 0 167m 50m 39m S 1 2.5 0:01.72 zmc
16353 www-data 20 0 123m 20m 11m S 1 1.0 0:03.29 zma
16396 www-data 20 0 122m 17m 8688 S 1 0.9 0:00.26 nph-zms
14997 www-data 20 0 30188 7184 3416 S 0 0.3 0:00.22 apache2
14999 www-data 20 0 30292 6916 3160 S 0 0.3 0:00.07 apache2
15000 www-data 20 0 30348 7276 3464 S 0 0.4 0:00.14 apache2
15001 www-data 20 0 29820 4304 1136 S 0 0.2 0:00.00 apache2
15013 www-data 20 0 30288 7004 3248 S 0 0.3 0:00.09 apache2
15014 www-data 20 0 30244 6844 3144 S 0 0.3 0:00.04 apache2
15015 www-data 20 0 30376 6872 3160 S 0 0.3 0:00.06 apache2
15016 www-data 20 0 30272 7016 3280 S 0 0.3 0:00.09 apache2
15662 www-data 20 0 30128 7080 3372 S 0 0.3 0:00.13 apache2
15663 www-data 20 0 30244 6844 3140 S 0 0.3 0:00.03 apache2
16326 www-data 20 0 10668 6100 1472 S 0 0.3 0:00.07 zmdc.pl
16361 www-data 20 0 13692 10m 2764 S 0 0.5 0:00.21 zmfilter.pl
16363 www-data 20 0 11140 7812 2744 S 0 0.4 0:00.17 zmaudit.pl
16366 www-data 20 0 10540 7308 2796 S 0 0.4 0:00.18 zmwatch.pl
16369 www-data 20 0 11780 8544 2752 S 0 0.4 0:00.19 zmupdate.pl
16392 www-data 20 0 122m 18m 9804 S 0 0.9 0:00.26 nph-zms
16394 www-data 20 0 122m 19m 10m S 0 0.9 0:00.31 nph-zms
16400 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2
16401 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2
16402 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2


### ### ### and, finally, apache processes and users under zm running during "hang state"
* apache error log detailed in the end during this state
: top -u www-data

top - 10:52:21 up 2:38, 1 user, load average: 0.06, 0.08, 0.02
Tasks: 153 total, 1 running, 152 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8%us, 0.4%sy, 0.0%ni, 98.7%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2069868k total, 942612k used, 1127256k free, 63832k buffers
Swap: 1646620k total, 0k used, 1646620k free, 604400k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16350 www-data 20 0 123m 20m 11m S 1 1.0 0:05.09 zma
16356 www-data 20 0 123m 20m 11m S 1 1.0 0:10.90 zma
16359 www-data 20 0 123m 20m 11m S 1 1.0 0:05.05 zma
16353 www-data 20 0 123m 20m 11m S 1 1.0 0:05.04 zma
16326 www-data 20 0 10668 6100 1472 S 0 0.3 0:00.10 zmdc.pl
16348 www-data 20 0 167m 50m 39m S 0 2.5 0:02.47 zmc
14997 www-data 20 0 30188 7184 3416 S 0 0.3 0:00.27 apache2
14999 www-data 20 0 30292 6916 3160 S 0 0.3 0:00.07 apache2
15000 www-data 20 0 30348 7276 3464 S 0 0.4 0:00.14 apache2
15013 www-data 20 0 30320 7080 3284 S 0 0.3 0:00.13 apache2
15015 www-data 20 0 30376 6872 3160 S 0 0.3 0:00.06 apache2
15016 www-data 20 0 30272 7016 3280 S 0 0.3 0:00.09 apache2
15662 www-data 20 0 30348 7228 3412 S 0 0.3 0:00.17 apache2
15663 www-data 20 0 30244 6860 3156 S 0 0.3 0:00.03 apache2
16361 www-data 20 0 13692 10m 2768 S 0 0.5 0:00.21 zmfilter.pl
16363 www-data 20 0 11140 7812 2744 S 0 0.4 0:00.17 zmaudit.pl
16366 www-data 20 0 10540 7308 2796 S 0 0.4 0:00.21 zmwatch.pl
16369 www-data 20 0 11780 8544 2752 S 0 0.4 0:00.19 zmupdate.pl
16396 www-data 20 0 122m 18m 9816 S 0 0.9 0:00.30 nph-zms
16400 www-data 20 0 30256 6904 3172 S 0 0.3 0:00.04 apache2
16401 www-data 20 0 30276 7020 3284 S 0 0.3 0:00.07 apache2
16402 www-data 20 0 31436 8104 3220 S 0 0.4 0:00.09 apache2
16734 www-data 20 0 122m 14m 5520 S 0 0.7 0:00.12 nph-zms
16735 www-data 20 0 122m 13m 4844 S 0 0.7 0:00.10 nph-zms
16737 www-data 20 0 122m 14m 5296 S 0 0.7 0:00.12 nph-zms
16747 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2
16748 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2
16749 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2

- We may see that the nph-zms processes remained active, and it´s now more than 5 minutes after my apache is hung; as we may see, they don´t want
to get closed by themselves; let´s give them some minutes and see what do we get again:

- wow, now something seems to have happened ! it looks like the zms´s were closed automatically (seems the zmaudit is a good boy), what is good,
but, apache is still hung, what is no good;


top - 11:00:00 up 2:46, 1 user, load average: 0.16, 0.08, 0.01
Tasks: 149 total, 1 running, 148 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.1%us, 0.5%sy, 0.0%ni, 98.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2069868k total, 1047220k used, 1022648k free, 65084k buffers
Swap: 1646620k total, 0k used, 1646620k free, 741772k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16350 www-data 20 0 123m 20m 11m S 1 1.0 0:10.93 zma
16356 www-data 20 0 123m 20m 11m S 1 1.0 0:20.28 zma
16359 www-data 20 0 123m 20m 11m S 1 1.0 0:10.82 zma
16353 www-data 20 0 123m 20m 11m S 1 1.0 0:10.82 zma
14997 www-data 20 0 30188 7184 3416 S 0 0.3 0:00.27 apache2
14999 www-data 20 0 30292 6916 3160 S 0 0.3 0:00.07 apache2
15000 www-data 20 0 30348 7276 3464 S 0 0.4 0:00.14 apache2
15013 www-data 20 0 30320 7080 3284 S 0 0.3 0:00.13 apache2
15015 www-data 20 0 30376 6872 3160 S 0 0.3 0:00.06 apache2
15016 www-data 20 0 30272 7016 3280 S 0 0.3 0:00.09 apache2
15662 www-data 20 0 30348 7228 3412 S 0 0.3 0:00.17 apache2
15663 www-data 20 0 30244 6860 3156 S 0 0.3 0:00.03 apache2
16326 www-data 20 0 10668 6100 1472 S 0 0.3 0:00.18 zmdc.pl
16348 www-data 20 0 167m 50m 39m S 0 2.5 0:04.96 zmc
16361 www-data 20 0 13692 10m 2768 S 0 0.5 0:00.21 zmfilter.pl
16363 www-data 20 0 11140 7812 2744 S 0 0.4 0:00.17 zmaudit.pl
16366 www-data 20 0 10540 7308 2796 S 0 0.4 0:00.28 zmwatch.pl
16369 www-data 20 0 11780 8544 2752 S 0 0.4 0:00.19 zmupdate.pl
16400 www-data 20 0 30256 6904 3172 S 0 0.3 0:00.04 apache2
16401 www-data 20 0 30276 7020 3284 S 0 0.3 0:00.07 apache2
16402 www-data 20 0 31436 8104 3220 S 0 0.4 0:00.09 apache2
16747 www-data 20 0 29820 4992 1828 S 0 0.2 0:00.00 apache2
16748 www-data 20 0 29820 4992 1828 S 0 0.2 0:00.00 apache2
16749 www-data 20 0 29760 3500 564 S 0 0.2 0:00.00 apache2



### ### ### lets have apache restarted and have a look once again:
: service apache2 restart
: top -u www-data

top - 11:06:32 up 2:52, 1 user, load average: 0.10, 0.07, 0.01
Tasks: 140 total, 2 running, 138 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.1%us, 0.2%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2069868k total, 1199164k used, 870704k free, 66600k buffers
Swap: 1646620k total, 0k used, 1646620k free, 907728k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16350 www-data 20 0 123m 20m 11m S 1 1.0 0:16.17 zma
16359 www-data 20 0 123m 20m 11m R 1 1.0 0:15.99 zma
16353 www-data 20 0 123m 20m 11m S 1 1.0 0:16.00 zma
16356 www-data 20 0 123m 20m 11m S 1 1.0 0:33.08 zma
16348 www-data 20 0 167m 50m 39m S 1 2.5 0:07.21 zmc
16326 www-data 20 0 10668 6100 1472 S 0 0.3 0:00.27 zmdc.pl
16361 www-data 20 0 13692 10m 2768 S 0 0.5 0:00.21 zmfilter.pl
16363 www-data 20 0 11140 7836 2748 S 0 0.4 0:00.18 zmaudit.pl
16366 www-data 20 0 10540 7308 2796 S 0 0.4 0:00.35 zmwatch.pl
16369 www-data 20 0 11780 8544 2752 S 0 0.4 0:00.19 zmupdate.pl
17557 www-data 20 0 29760 3496 564 S 0 0.2 0:00.00 apache2
17558 www-data 20 0 29760 3496 564 S 0 0.2 0:00.00 apache2
17559 www-data 20 0 29760 3496 564 S 0 0.2 0:00.00 apache2
17560 www-data 20 0 29760 3496 564 S 0 0.2 0:00.00 apache2
17561 www-data 20 0 29760 3496 564 S 0 0.2 0:00.00 apache2


-- and that´s it; we now have apache restarted, and no need to restart zm or anything else;
-- note that zm didn´t stop to work or capture for any moment, it´s just a question of being able (or not) to access the system from the web
server/interface;


### ### ### apache´s error log during hang: (it seems to be a useful information)

[Thu May 07 11:59:29 2009] [error] [client 189.83.71.235] socket_sendto( /tmp/zms-180692s.sock ) failed: No such file or directory, referer:
http://xxxundisclosedxxx.no-ip.biz:9901 ... ge&group=0
[Thu May 07 11:59:29 2009] [error] [client 189.83.71.235] array (\n 0 => \n array (\n 'file' => '/var/www/ajax/stream.php',\n 'line' =>
51,\n 'function' => 'ajaxError',\n 'args' => \n array (\n 0 => 'socket_sendto( /tmp/zms-180692s.sock ) failed: No such file or
directory',\n ),\n ),\n 1 => \n array (\n 'file' => '/var/www/index.php',\n 'line' => 116,\n 'args' => \n array (\n 0 =>
'/var/www/ajax/stream.php',\n ),\n 'function' => 'require_once',\n ),\n), referer:
http://xxxundisclosedxxx.no-ip.biz:9901 ... ge&group=0
[Thu May 07 11:59:44 2009] [error] [client 189.83.71.235] socket_sendto( /tmp/zms-323704s.sock ) failed: No such file or directory, referer:
http://xxxundisclosedxxx.no-ip.biz:9901 ... ge&group=0
[Thu May 07 11:59:44 2009] [error] [client 189.83.71.235] array (\n 0 => \n array (\n 'file' => '/var/www/ajax/stream.php',\n 'line' =>
51,\n



### ### ### list of sock files during hang: (it seems to be a useful information, lots of sock files exist when no stream is in fact working)

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-161907s.sock zms-305993s.sock zms-446957s.sock zms-683243w.sock
zms-037043s.sock zms-161907w.sock zms-309546w.sock zms-573633s.sock zms-815284w.sock
zms-139770s.sock zms-291443s.sock zms-401992w.sock zms-675610s.sock zms-816487w.sock


### ### ### piece of zm_debug.log.xxxx during hang: (doesnt seem to be usefull, because looks like the same when system works correctly)


05/07/09 11:59:33.493141 zms[21051].INF-zm_debug.c/292 [New Debug Level = 9, New Debug Log = /tmp/zm_debug.log.21051]
05/07/09 11:59:33.493278 zms[21051].DB1-zms.cpp/94 [Query: mode=jpeg&monitor=1&scale=100&maxfps=15&buffer=1000&connkey=604143&rand=1241708373]
05/07/09 11:59:33.493499 zms[21051].DB1-zm_monitor.cpp/2264 [Got 1 monitors]
05/07/09 11:59:33.494052 zms[21051].DB1-zm_monitor.cpp/338 [monitor purpose=0]
05/07/09 11:59:33.494067 zms[21051].DB1-zm_monitor.cpp/345 [mem.size=9217192]
05/07/09 11:59:33.494165 zms[21051].DB1-zm_zone.cpp/50 [Initialised zone 0/All - 1 - 320x240 - Rgb:ff0000, CM:3, MnAT:15, MxAT:0, MnAP:50,

MxAP:75000, FB:3x3, MnFP:50, MxFP:50000, MnBS:10, MxBS:0, MnB:0, MxB:0, OF: 0]
05/07/09 11:59:33.494273 zms[21051].DB9-zm_image.cpp/1354 [x1:0,y1:239 x2:0,y2:0]
05/07/09 11:59:33.494288 zms[21051].DB9-zm_image.cpp/1354 [x1:0,y1:0 x2:319,y2:0]
05/07/09 11:59:33.494300 zms[21051].DB9-zm_image.cpp/1354 [x1:319,y1:0 x2:319,y2:239]
05/07/09 11:59:33.494311 zms[21051].DB9-zm_image.cpp/1354 [x1:319,y1:239 x2:0,y2:239]
05/07/09 11:59:33.494326 zms[21051].DB9-zm_image.cpp/1374 [0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494354 zms[21051].DB9-zm_image.cpp/1374 [1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]
05/07/09 11:59:33.494369 zms[21051].DB9-zm_image.cpp/1388 [Moving global edge]
05/07/09 11:59:33.494383 zms[21051].DB9-zm_image.cpp/1388 [Moving global edge]
05/07/09 11:59:33.494394 zms[21051].DB9-zm_image.cpp/1409 [0 - 0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494407 zms[21051].DB9-zm_image.cpp/1409 [0 - 1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]
05/07/09 11:59:33.494424 zms[21051].DB9-zm_image.cpp/1409 [1 - 0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494437 zms[21051].DB9-zm_image.cpp/1409 [1 - 1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]
05/07/09 11:59:33.494453 zms[21051].DB9-zm_image.cpp/1409 [2 - 0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494466 zms[21051].DB9-zm_image.cpp/1409 [2 - 1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]
05/07/09 11:59:33.494482 zms[21051].DB9-zm_image.cpp/1409 [3 - 0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494495 zms[21051].DB9-zm_image.cpp/1409 [3 - 1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]
05/07/09 11:59:33.494511 zms[21051].DB9-zm_image.cpp/1409 [4 - 0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494524 zms[21051].DB9-zm_image.cpp/1409 [4 - 1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]
05/07/09 11:59:33.494540 zms[21051].DB9-zm_image.cpp/1409 [5 - 0: min_y: 0, max_y:239, min_x:0.00, 1/m:-0.00]
05/07/09 11:59:33.494552 zms[21051].DB9-zm_image.cpp/1409 [5 - 1: min_y: 0, max_y:239, min_x:319.00, 1/m:0.00]



### ### ### finally, lets have a look at the sock files (that called my attention in /tmp) from a clean reboot/start of the rig until the hang
state:

1- zm started state after clean reboot: (one single zmdc.sock)
root@c2qhome:/tmp# ls *.sock
zmdc.sock


2- zm streaming montage view correctly with 4 analog cameras on 320x240: (one zms-xxxxxxs.sock for each camera)
root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-604428s.sock zms-910338s.sock
zms-600548s.sock zms-648188s.sock
root@c2qhome:/tmp#


3- now i have closed the montage view before any "hang", and the zms-xxxxxx.sock files disappeared automatically, looks good so far !
root@c2qhome:/tmp# ls *.sock
zmdc.sock
root@c2qhome:/tmp#


4- now we have zm under apache "pseudo-hang state": (as i try to reproduce the error to get apache into "hang", i notice my sock files start to

grow, from 4, to 5, to 6, and now they don´t seem to close automatically as i think they should...)
root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-145650s.sock zms-442062s.sock zms-836073s.sock
zms-011276s.sock zms-410670s.sock zms-599168s.sock

5- i continue to force the system hang, and it seem i have success on that!
root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-145650s.sock zms-593226s.sock zms-836073s.sock
zms-011276s.sock zms-410670s.sock zms-599168s.sock zms-929479s.sock
zms-139301s.sock zms-442062s.sock zms-686860s.sock

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-145650s.sock zms-410670s.sock zms-599168s.sock
zms-011276s.sock zms-371325s.sock zms-442062s.sock zms-789924s.sock
zms-139301s.sock zms-386420s.sock zms-593226s.sock zms-836073s.sock

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-244402s.sock zms-534589s.sock zms-909587s.sock
zms-020641s.sock zms-371325s.sock zms-593226s.sock zms-909587w.sock
zms-139301s.sock zms-386420s.sock zms-635070s.sock
zms-159964s.sock zms-468255s.sock zms-789924s.sock

*** At this exact moment, my firefox start to wait forever and made me thing that apache was hung, so now lets forget ff and open a montage view
from IE, and see what happens:

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-166020s.sock zms-468255s.sock zms-789924s.sock
zms-020641s.sock zms-166020w.sock zms-534589s.sock zms-821899s.sock
zms-032397s.sock zms-244402s.sock zms-593226s.sock zms-909587s.sock
zms-139301s.sock zms-371325s.sock zms-635070s.sock zms-909587w.sock
zms-159964s.sock zms-386420s.sock zms-693534s.sock

* do you see ? now we have more socks due to IE, when FF is still waiting forever....
Now i´ll try something: i´ll be removing the all the sock files by hand:

root@c2qhome:/tmp# rm -rf ./*.sock
root@c2qhome:/tmp# ls *.sock
ls: cannot access *.sock: No such file or directory

* guess what: no luck, i didn´t have it working back on FF, (even trying f5, ctrl-f5), BUT IE is still streaming perfectly.... after a short while zm stops

for itself, probably because i have deleted a sock file that i shouldnt have to; so i start zm again from IE, and at this point a new zmdc.sock
appears under /tmp, zm starts normally again, but my FF session is still dead; with no other resource, i close all instances of FF and restart it,
then i have under streaming:

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-395807s.sock zms-716693s.sock
zms-247777s.sock zms-429095s.sock


now i close montage view on firefox, wait a little while, and i think the sock files will dissapear, but look:

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-247777s.sock zms-429095s.sock


I have now two orphaned sock files that i think shouldn´t be there; waited for minutes and they remain there;
I then open another montage, and then

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-429095s.sock zms-694568s.sock zms-894674s.sock
zms-247777s.sock zms-551900s.sock zms-758094s.sock


As you see, socks starts to acumulate again..... thats the deadline for me ! I really think the audit process that killed inactive zms processes
may end up solving this when it also corrects this sock files or session hang, in one way or another; also as i have mentioned, an "elegant" close
of the socks instead of the javascript closewindow event look to me as a possibility;

in the end, my firefox CRASHED with:

Add-ons: {CAFEEFAC-0016-0000-0013-ABCDEFFEDCBA}:6.0.13,jqs@sun.com:1.0,{972ce4c6-7e08-4474-a285-3208198ce6fd}:3.0.10
BuildID: 2009042316
CrashTime: 1241723399
InstallTime: 1240928037
ProductName: Firefox
SecondsSinceLastCrash: 282878
StartupTime: 1241723156
Theme: classic/1.0
URL: http://xxxundisclosedxxxx.no-ip.biz:990 ... ge&group=0
Vendor: Mozilla
Version: 3.0.10

and, just to mention, if I leave IE6 waiting too long on the montage view, it also crashes with evidence of memory corruption on the client side;
"memory cant be read" warning;


Really sorry for this large post, but on the "README BEFORE POSTING! Troubleshooting" i found that i should include pieces of useful logs, what i
exactly tried to do;

TKS once again for any attention and support on that; as i mentioned, restarting FF seems to be a workaround, but sock files acumulating over and
over look to me as a remaining fault, what im not sure it is;

[rdmelin]

henriquejf, would you try applying mitch's patch and see if this fixes your troubles.
http://www.zoneminder.com/forums/viewtopic.php?t=13615

Also you may want to tweak your firefox setting per this note in the FAQ
http://www.zoneminder.com/wiki/index.ph ... FireFox.3F

[henriquejf]

henriquejf, would you try applying mitch's patch and see if this fixes your troubles.

right away, please hold !

[henriquejf]

henriquejf, would you try applying mitch's patch and see if this fixes your troubles.
http://www.zoneminder.com/forums/viewtopic.php?t=13615

Well, no success unfortunately; i used the wget http://mitchcapper.com/zm_web_fixes.patch, recompiled, recontructed my db from zero, reconstructed my 4 camera configs, got the rig rebooted and working just as before; in no more than 3 tryes, i had ff stopped responding; (actually, i force this sittuation upon opening the montage, closing, opening again, etc...)

it looked to me the problem remained pretty the same; with next to zero effort i could observe how the sock files grown and (seem) to led to the situation, look:

root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-617997s.sock zms-674314s.sock zms-827856s.sock
root@c2qhome:/tmp#
root@c2qhome:/tmp#
root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-617997s.sock zms-674314s.sock zms-827856s.sock
root@c2qhome:/tmp#
root@c2qhome:/tmp#
root@c2qhome:/tmp#
root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-335228s.sock zms-659592s.sock zms-827856s.sock
zms-089692s.sock zms-573613s.sock zms-668461s.sock zms-980774s.sock
zms-215930s.sock zms-617997s.sock zms-674314s.sock
root@c2qhome:/tmp#
root@c2qhome:/tmp# ls *.sock
zmdc.sock zms-573613s.sock zms-824260s.sock zms-993027s.sock
zms-335228s.sock zms-617997s.sock zms-827856s.sock
zms-530289w.sock zms-674314s.sock zms-980774s.sock
root@c2qhome:/tmp#


at this point, FF is waiting forever, and browsing from IE or restarting FF make things work again;

tonight ill be trying to install newvisionantenna´s livecd, update it from svn, and make my prays; if i observe the situation again or have it working as expected, i´ll let you guys know !

TKS for your time !!!

oh, and

Also you may want to tweak your firefox setting per this note in the FAQ

Yes, i have tryed that, have noticed performance improve, but system gets more unstable (crashes FF 3.0.10);

[henriquejf]

I wish i did not succeed, but i did manage to reproduce the error in a new fresh install of zm dvd (debian enchillada 1.24.1 md5 06b01c18dbd4d1cf38e1a6c89704a76a ); in short, its like that

* i cant reproduce the error on the lan or localhost of zm;

* i CAN reproduce the error easily on wan side; i start to load a montage view of 4 cameras, hit esc to stop, hit f5, hit esc, (...) and then i start to observe how zm sock files start to grow on /tmp; (at this moment the browser is not able to communicate with the web gui anymore)

* This does not happen on the lan side, because i notice that at the exact time i stop loading the montage view, all zm streaming sock disappear of /tmp at the same time ! so i think the behavior is 100% correct under lan/localhost, but on the wan side, look below: (state of /tmp after some trying)

root@amdcctvzmdvd:/tmp# ls
gconfd-chuckoms zmdc.log zms-293234s.sock zms-718792s.sock
gpg-9nBp90 zmdc.sock zms-354928s.sock zms-743213s.sock
kde-chuckoms zmfilter.log zms-417767s.sock zms-857899s.sock
kde-root zmpkg.log zms-445646s.sock zms-873150s.sock
ksocket-chuckoms zms-071949s.sock zms-505254s.sock zmupdate.log
orbit-chuckoms zms-227598s.sock zms-535112s.sock zmwatch.log
ssh-YVXjNZ6500 zms-231050s.sock zms-535112w.sock
tmp.cdoSjR6309 zms-236615s.sock zms-648730s.sock
zmaudit.log zms-288122s.sock zms-702250s.sock
root@amdcctvzmdvd:/tmp#


at this point, the browser starts to wait forever and restarting is needed; as i said, at the localhost the socks seam to dissapear the exact moment i hit esc on the browser;

I dont want to be bottering anyone, zm is a really great software and the community makes a great work, im just trying to expose a situation that i can reproduce along different versions and zm distros so far;

this last one was upon a fresh install of zm dvd 1.24.1; now ill try to update it against trunk/svn and post comments back;

If someone is interested, i can try to help reproduce the error on other machines, just give me access to the montage view (at least 4 cameras), dont need any access to options; then i need the machine owner to observe the /tmp directory as i try it; the moment i succeed, it will be possible to see many sock files that shouldnt be there in /tmp; (i dont bet any money i will get it, and please once again consider as a improving opportunity, it is no criticism at ALL)

[newvisionantenna]

I think I'm following what your saying. I'll have to try it out here.

Also, try what your saying here,

67.210.200.98/zm and login with admin/admin

That is the latest, as of yesterday SVN version of Zoneminder. I updated by hand last night, see what you get there. If you need more cam's on it let me know and I'll add them. If the problem is fixed then I will take the time to make an update, but if it's not then really no need to worry about it for now. The DVD has the SVN/Trunk version prior to yesterday's release.

I'm using almost always IE7/8 and never had any problems watching the montage view. I'm not sure what your saying about hit esc and f5, why would you be doing that combo of button pressing? Just curious, trying to understand. I've also moved away from the montage view and pretty much use jZMconsole on my Vista desktop. The only thing I'm waiting for is an update on that program that can better adjust the picture size when using it so 640x480 does not take up a good portion of my screen.

[newvisionantenna]

It's probably best if I let you try and reproduce the error. I looked at the info your saying and couldn't reproduce it. I'm going to the IP above and opened montage, hitting esc and refreshing it. I also closed and opened it about 25 times and each time it came right back up. It only has 3 cam's right now so I'm not sure if it needs more to have the error occur.

[henriquejf]

It's probably best if I let you try and reproduce the error.

ill be out for lunch for 2 hours now and then ill try to reproduce the behavior on your server; as a matter of fact, i am not crazy trying to stop/reload/stop/reload the montage view; what i have is a situation that leaving the montage open (or even the web interface) gives me this hang, and i found a way to reproduce it faster that way;

as i have mentioned on other posts, i simply close the browser, reopen it and then i got it working back, and i also have the options of the clients too, but i still want to investigate it deeper, and maybe contribute to zm project in a way; other users experienced the same problem, as they opened the threads, not me; but actually, if i think i am making more noise then helping, and if other users dont still have this situation, ill let it beside as if it never happened;

ill be trying that on your updated server after lunch !

bad luck for me so that i dont succeed !!!!

[newvisionantenna]

I think your helping, people have been talking about this issue for awhile now. Hopefully it can be fixed. I've just never had the issue yet, but maybe I'm not leaving it on long enough or something. Each person uses ZM different so maybe the way I'm using it never shows the error. Plus I've started using jZMConsole so I rarely look at the web page any more. I can leave it running for days and never have it crash. It just needs alittle more work put into it and it could be a really slick application. The ZMviewer is also another tool, but it's pretty much limited to running on the machine.

[henriquejf]

got it ! unfortunately/fortunately;

i can reproduce the error on your server, it was exactly 18:32 gmt 11/may/2009;

i have marked your apache error log with this line: just seconds after reproducing the error;


http://67.210.200.98/zm/--------------- ... ----------

i can bet that your tmp dir has more zms socks than it should, and that some important information may have appeared on your error log before my mark;

if you want to watch it on real time, ask me again so you can watch /tmp, because after some minutes the sock files dissapear gradually;

i was using ip 189.83.39.252, you´ll find it for sure;

Tks for your attention !!!


edit:: newvissionantenna just captured what i was talking about:

zms-001661s.sock zms-343885s.sock zms-650246s.sock zms-920818s.sock
zms-001786s.sock zms-353389s.sock zms-650830s.sock zms-940104s.sock
zms-011612s.sock zms-364180s.sock zms-651385s.sock zms-963132s.sock
zms-013580s.sock zms-371508s.sock zms-661078s.sock zms-967539s.sock
zms-023461s.sock zms-381108s.sock zms-686013s.sock zms-998702s.sock
zms-035217s.sock zms-385301s.sock zms-692595s.sock zmswap-m1
zms-055851s.sock zms-405052s.sock zms-696124w.sock zmswap-m11
zms-061125s.sock zms-417921s.sock zms-703670s.sock zmswap-m14
zms-067041s.sock zms-428596s.sock zms-720235s.sock zmswap-m16
zms-079950s.sock zms-451451s.sock zms-728453s.sock zmswap-m17
zms-082377s.sock zms-452588s.sock zms-731558s.sock zmswap-m18
zms-095096s.sock zms-472490s.sock zms-732084s.sock zmswap-m19
zms-122867s.sock zms-481601w.sock zms-735462s.sock zmswap-m2
zms-164842s.sock zms-493429s.sock zms-751427s.sock zmswap-m3
zms-167790s.sock zms-512754s.sock zms-760326s.sock zmswap-m5
zms-181379s.sock zms-534101s.sock zms-762824s.sock zmupdate.log
zms-193486s.sock z

wow !

so that is that ! please also look for my mark on your error.log and give me any related info before my mark; im positive there are info regarding socks that can not be found in it !!!


edit: Aaron has just provided me with the errors i was sure he had in his error.log:

[Mon May 11 17:17:18 2009] [error] [client 189.83.39.252] array (\n 0 => \n array (\n 'file' => '/var/www/zm/ajax/stream.php',\n 'line' => 55
[Mon May 11 17:17:21 2009] [error] [client 189.83.39.252] socket_sendto( /tmp/zms-377808s.sock ) failed: No such file or directory, referer: http://6
[Mon May 11 17:17:21 2009] [error] [client 189.83.39.252] array (\n 0 => \n array (\n 'file' => '/var/www/zm/ajax/stream.php',\n 'line' => 55
[Mon May 11 17:17:26 2009] [error] [client 189.83.39.252] socket_sendto( /tmp/zms-456136s.sock ) failed: No such file or directory, referer: http://6
[Mon May 11 17:17:26 2009] [error] [client 189.83.39.252] array (\n 0 => \n array (\n 'file' => '/var/www/zm/ajax/stream.php',\n 'line' => 55
[Mon May 11 17:19:51 2009] [error] [client 189.83.39.252] socket_sendto( /tmp/zms-671014s.sock ) failed: No such file or directory, referer: http://6
[Mon May 11 17:19:51 2009] [error] [client 189.83.39.252] array (\n 0 => \n array (\n 'file' => '/var/www/zm/ajax/stream.php',\n 'line' => 55
[Mon May 11 17:22:09 2009] [error] [client 189.83.39.252] /usr/lib/cgi-bin/nph-zms: error while loading shared libraries: cannot create cache for sea
[Mon May 11 17:22:41 2009] [error] [client 189.83.39.252] socket_sendto( /tmp/zms-557096s.sock ) failed: No such file or directory, referer: http://6
[Mon May 11 17:22:41 2009] [error] [client 189.83.39.252] array (\n 0 => \n array (\n 'file' => '/var/www/zm/ajax/stream.php',\n 'line' => 55
[Mon May 11 17:22:55 2009] [error] [client 189.83.39.252] socket_sendto( /tmp/zms-557096s.sock ) failed: No such file or directory, referer: http://6
[Mon May 11 17:22:55 2009] [error] [client 189.83.39.252] array (\n 0 => \n array (\n 'file' => '/var/www/zm/ajax/stream.php',\n 'line' => 55
[Mon May 11 17:23:17 2009] [error] [client 189.83.39.252] socket_sendto( /tmp/zms-886486s.sock ) failed: No such file or directory, referer: http:/




Tks so much, if you want you may remove the cameras now; i also took a printscreen showing firefox hung while from IE i could still access the montage;

in the end, i wasnt (absolutely) crazy;

it took me 3 tryes to reproduce the error when you let 3 cameras for me to play; with only one camera (tripcheck.com) i could not reproduce the error; so it makes me think that it is related to multiple streams; duh!!!

[newvisionantenna]

I captured a screen shot of it, there was a lot of sock's in there. I wish I new more on how to address it but it looks like something with Zoneminder itself. I've never saw the issue till you just showed me. Maybe I don't leave the montage up long enough or something. I've also been using zmviewer or jzmconsole lately unless I need to edit settings. I find the other two options better and since I can use it on windows that's pretty much what I do lately. I know it's important that the ZM montage and viewing get's fixed, so as soon as it does I'll put my effort into rolling it into an installable DVD or update. Thanks for your help and continued work on this issue.

Aaron

[henriquejf]

Ok Aaron !!! tks for your time and patience; ::EDIT:: i think it may be a browser bug;

in the best intention of helping, i offer to TRY to reproduce the error on anyones server, count on me from 09am to 12pm and from 14pm to 18pm brazil local time, that is

gmt 13pm to 16pm and 18pm to 22pm

count on me anyone !

just send me a pm; if needed i can be online on other time also;

::EDIT:: The key to reproduce this error is the "lag" between the zm box and the client; i can´t manage to reproduce it on the lan side, so i think that is why some people were not able to reproduce it; it still hapens with ffmpeg svn revision 29318/18890 and zm svn revision 2887, following my ubuntu 9.04 install guide at
http://www.zoneminder.com/wiki/index.ph ... rver_32bit

also, please note: when you watch a stream with FF and stop the stream, some zms socks are left behind for a while; under Chrome, when you close the stream, the socks disappear immediately; it does not seem to be zm fault then.....

[maufacc]

somthing to do with ZM..
after 10 hours uptime i have this
"
[root@zmhost shm]# cat /proc/sys/fs/file-nr
40704 0 50837
"
then i do this
[root@zmhost shm]# /etc/rc.d/zm stop
:: Stopping ZoneMinder [DONE]
[root@zmhost shm]# cat /proc/sys/fs/file-nr
896 0 50837

and my open files come down to 896 ???

why ? any1 knows ?

thanks

Following with your work, I start killing each process related with zm, and i've found that killing zmwatch.pl, the allocated file descriptors goes to the correct level.
As I can see something inside that pl do not deallocate the file descriptors. I do not understand too much perl.
I do not have enough knowledge to patch.
I am using zm 1.24.2