Serious Slowdown

jameswilson · Post by **jameswilson** » Mon Dec 05, 2005 5:38 pm

My biggest system has just started having serious performance issues and i cant find anything wrong with it.
It has 16 composite cams and 5 ip cams on it

It used to be setup ofr mocord on all channels
2 fps on all inpputs
would ussualyy sit at a load of about 1-2 with a 50% idle on the processor.

Now i have 0.02 % idle and loadds of 20+

I have turned all inputs to record and dropped to 1 fps but nothing doing.
I have checked mysql is ok. Im not dropping events but logging into console can take 15 minutes.
Its setup on 4 x 250Gb drives (raid 0) and im wondering if i have an issue here but i have no error in my logs. I have 2gig of ram in it so its unlikely to be ram.
I have no stability issues its just damn slow, and stalls when watching live images.

Any ideas Guys

Post by **zoneminder** » Mon Dec 05, 2005 6:50 pm

If you run 'top' what is it that sits at the 'top'?

Phil

jameswilson · Post by **jameswilson** » Mon Dec 05, 2005 6:56 pm

on eof the ip cams zmc -m 20
using a whole 6.4% of proc
I have tried the obvious things
Th eonly thing i can think of is i enabled bandwidth monitoring and things went bad about 2 months ago, I have just disabled all the firewalls on it in case its having a problem writing all that to syslog. But i am as they say seriously clutching now lol

The top few thing in top are all zmc's and there taking between 2 and 6%

I have just flushed all the firewall configs in the vain hope etc etc

What i dont understand is all was great on it, i have now limited all cams to 1 fps and put them all on record, it helps but i have a duron 750 with 9 cams on it that is close to its limit but this thing shouldnt be
3.4Gig P4 with lots of fans in it so im pretty sure its not to hot

The only other stray i can grab at is the proc fan has failed but id expect it to die if that happened (oh and this machine is remote at the mo)

James

Post by **zoneminder** » Mon Dec 05, 2005 7:12 pm

What are the other load average values over time? It's worth watching for a while as well as load average can be influenced by something diving in and using up a bunch of cpu for a few seconds and then disappearing. The most likely candidate if it is degrading is mysql, you can set up slow query (or all query logging) on mysql to check. In which case increasing the period between runs of zmfilter and zmaudit can help.

Phil

jameswilson · Post by **jameswilson** » Mon Dec 05, 2005 7:16 pm

they are all high usually 15 min is about 15
I can get the load right down now to about 6

After flushing those firewall rules

Code: Select all

Display :    PID  User  Memory  CPU  Search  Run..  
CPU load averages: 14.76 (1 mins) , 14.83 (5 mins) , 11.59 (15 mins)

Process ID Owner CPU Command 
2860 apache 6.2 % /usr/bin/zmc -m 20 
2840 apache 6.1 % /usr/bin/zmc -m 18 
2850 apache 5.9 % /usr/bin/zmc -m 19 
3282 apache 3.1 % /usr/bin/zma -m 16 
3289 apache 2.9 % /usr/bin/zma -m 12 
3283 apache 2.8 % /usr/bin/zma -m 7 
3288 apache 2.7 % /usr/bin/zma -m 9 
3293 apache 2.7 % /usr/bin/zma -m 6 
3277 apache 2.5 % /usr/bin/zma -m 13 
3300 apache 2.5 % /usr/bin/zma -m 8 
3310 apache 2.5 % /usr/bin/zma -m 15 
14596 root 2.5 % /usr/libexec/webmin/proc/index_cpu.cgi 
2782 apache 2.4 % /usr/bin/zma -m 10 
3271 apache 2.4 % /usr/bin/zma -m 11 
3308 apache 2.4 % /usr/bin/zma -m 14 
2870 apache 2.3 % /usr/bin/zmc -m 23 
2875 apache 2.3 % /usr/bin/zmc -m 24 
9906 apache 2.1 % /usr/bin/zma -m 3 
10570 apache 1.7 % /usr/bin/zma -m 5 
2777 apache 1.5 % /usr/bin/zmc -d 1 
2811 apache 1.5 % /usr/bin/zmc -d 2

But i just enabled mocord on the first 4 inputs and this happened

I have turned them back to record now and im hoping it will calm down again

jameswilson · Post by **jameswilson** » Tue Dec 06, 2005 9:56 am

im out nowon this i think its gonna need a rebuild. Load is ave at 15-18 with 15 composite cams at 1fps and 3 ip cams at 2 fps. All in record mode. Its working fine but its not happy about something and i dont know what. I thought it might be the number of events it had as i seem to have a lot of mysql processes (over 20) running and the purghe when full was set at 99% i have changed this to 95% with no improvemnt as yet (and disk is at 95%) Anyone any ideas on what to check as i reaaly dont want to rebuild this

James

Post by **zoneminder** » Tue Dec 06, 2005 10:03 am

You could try adding slow query logging to mysql. Edit /etc/my.cnf (warning may be different on your system!) and add the following lines under the [mysqld] section

Code: Select all

[mysqld]
log-slow-queries
set-variable=long_query_time=2

This will log all queries that take longer than 2 seconds to a file in /var/lib/mysql (usually). It may not reveal anything but it's worth a look. Some queries always take a long time so expect some entries in there, the interesting part will be to see how many. Also, of course logging will slow things down a touch itself so turn it off when you are done.

Phil

jameswilson · Post by **jameswilson** » Tue Dec 06, 2005 10:08 am

ok i will give that a go, im not getting any mysql processes take large amount of proc thought nbut i do have the system taking about 18% whatever that means

jameswilson · Post by **jameswilson** » Tue Dec 06, 2005 3:51 pm

still struggling here

Is there a way to see whats cauding the high queue ie whats in it a snapshot in time

I only have webmin access to this box mind

jameswilson · Post by **jameswilson** » Tue Dec 06, 2005 11:11 pm

I have tried everything i can think of and i cant get the load back to how it used to be, it used to have 30 - 30% idle time on it now its maxed

Im wondering if i have a bad stick of ram but i would expect it to fall over, i also remember enabling HT i might try disabling this unles anyone else has any suggestions

James

victor_diago · Post by **victor_diago** » Tue Dec 06, 2005 11:30 pm

Hi james.

Did it reached the maximum percent usage ?

It could be cleaning the disk and that surelly cause some slowdown on the server.

Victor Diago

jameswilson · Post by **jameswilson** » Tue Dec 06, 2005 11:33 pm

yeah but its conatantly really slow. It used to sit at about a load of 1-2 maybe peak at 3 but now its peaking at 25 and it takes 20mins (homestly) to log intot he web interface

victor_diago · Post by **victor_diago** » Tue Dec 06, 2005 11:33 pm

Hi james.

Did it reached the maximum percent usage ?

It could be cleaning the disk and that surelly cause some slowdown on the server.

Victor Diago

victor_diago · Post by **victor_diago** » Tue Dec 06, 2005 11:44 pm

Yeah, i have read again the things you said.

PLease see how many events do you have... i had problems dealing with a lot of events just right you said ... console takes a huge time to appear, thtas because mysql have to count all entries many times. that cause a lot of cpu utilization and the delay to open the console.

You could disable some queries in the console to make it faster.

Another thing that could cause cpu utilization is network monitoring, mainly if you are watching by high fps.

Wish this helps

Victor Diago

Post by **zoneminder** » Wed Dec 07, 2005 1:14 pm

If you can bear it, you could try deleting a lot (or even all) of your events and seeing if that makes a difference. That will at least give some indication of whether it is DB related.

Also try turning off zmaudit and zmfilter manually, do

Code: Select all

zmdc.pl stop zmfilter.pl

and the same for zmaudit.pl

Phil