Page 1 of 6
Serious Slowdown
Posted: Mon Dec 05, 2005 5:38 pm
by jameswilson
My biggest system has just started having serious performance issues and i cant find anything wrong with it.
It has 16 composite cams and 5 ip cams on it
It used to be setup ofr mocord on all channels
2 fps on all inpputs
would ussualyy sit at a load of about 1-2 with a 50% idle on the processor.
Now i have 0.02 % idle and loadds of 20+
I have turned all inputs to record and dropped to 1 fps but nothing doing.
I have checked mysql is ok. Im not dropping events but logging into console can take 15 minutes.
Its setup on 4 x 250Gb drives (raid 0) and im wondering if i have an issue here but i have no error in my logs. I have 2gig of ram in it so its unlikely to be ram.
I have no stability issues its just damn slow, and stalls when watching live images.
Any ideas Guys
Posted: Mon Dec 05, 2005 6:50 pm
by zoneminder
If you run 'top' what is it that sits at the 'top'?
Phil
Posted: Mon Dec 05, 2005 6:56 pm
by jameswilson
on eof the ip cams zmc -m 20
using a whole 6.4% of proc
I have tried the obvious things
Th eonly thing i can think of is i enabled bandwidth monitoring and things went bad about 2 months ago, I have just disabled all the firewalls on it in case its having a problem writing all that to syslog. But i am as they say seriously clutching now lol
The top few thing in top are all zmc's and there taking between 2 and 6%
I have just flushed all the firewall configs in the vain hope etc etc
What i dont understand is all was great on it, i have now limited all cams to 1 fps and put them all on record, it helps but i have a duron 750 with 9 cams on it that is close to its limit but this thing shouldnt be
3.4Gig P4 with lots of fans in it so im pretty sure its not to hot
The only other stray i can grab at is the proc fan has failed but id expect it to die if that happened (oh and this machine is remote at the mo)
James
Posted: Mon Dec 05, 2005 7:12 pm
by zoneminder
What are the other load average values over time? It's worth watching for a while as well as load average can be influenced by something diving in and using up a bunch of cpu for a few seconds and then disappearing. The most likely candidate if it is degrading is mysql, you can set up slow query (or all query logging) on mysql to check. In which case increasing the period between runs of zmfilter and zmaudit can help.
Phil
Posted: Mon Dec 05, 2005 7:16 pm
by jameswilson
they are all high usually 15 min is about 15
I can get the load right down now to about 6
After flushing those firewall rules
Code: Select all
Display : PID User Memory CPU Search Run..
CPU load averages: 14.76 (1 mins) , 14.83 (5 mins) , 11.59 (15 mins)
Process ID Owner CPU Command
2860 apache 6.2 % /usr/bin/zmc -m 20
2840 apache 6.1 % /usr/bin/zmc -m 18
2850 apache 5.9 % /usr/bin/zmc -m 19
3282 apache 3.1 % /usr/bin/zma -m 16
3289 apache 2.9 % /usr/bin/zma -m 12
3283 apache 2.8 % /usr/bin/zma -m 7
3288 apache 2.7 % /usr/bin/zma -m 9
3293 apache 2.7 % /usr/bin/zma -m 6
3277 apache 2.5 % /usr/bin/zma -m 13
3300 apache 2.5 % /usr/bin/zma -m 8
3310 apache 2.5 % /usr/bin/zma -m 15
14596 root 2.5 % /usr/libexec/webmin/proc/index_cpu.cgi
2782 apache 2.4 % /usr/bin/zma -m 10
3271 apache 2.4 % /usr/bin/zma -m 11
3308 apache 2.4 % /usr/bin/zma -m 14
2870 apache 2.3 % /usr/bin/zmc -m 23
2875 apache 2.3 % /usr/bin/zmc -m 24
9906 apache 2.1 % /usr/bin/zma -m 3
10570 apache 1.7 % /usr/bin/zma -m 5
2777 apache 1.5 % /usr/bin/zmc -d 1
2811 apache 1.5 % /usr/bin/zmc -d 2
But i just enabled mocord on the first 4 inputs and this happened
I have turned them back to record now and im hoping it will calm down again
Posted: Tue Dec 06, 2005 9:56 am
by jameswilson
im out nowon this i think its gonna need a rebuild. Load is ave at 15-18 with 15 composite cams at 1fps and 3 ip cams at 2 fps. All in record mode. Its working fine but its not happy about something and i dont know what. I thought it might be the number of events it had as i seem to have a lot of mysql processes (over 20) running and the purghe when full was set at 99% i have changed this to 95% with no improvemnt as yet (and disk is at 95%) Anyone any ideas on what to check as i reaaly dont want to rebuild this
James
Posted: Tue Dec 06, 2005 10:03 am
by zoneminder
You could try adding slow query logging to mysql. Edit /etc/my.cnf (warning may be different on your system!) and add the following lines under the [mysqld] section
Code: Select all
[mysqld]
log-slow-queries
set-variable=long_query_time=2
This will log all queries that take longer than 2 seconds to a file in /var/lib/mysql (usually). It may not reveal anything but it's worth a look. Some queries always take a long time so expect some entries in there, the interesting part will be to see how many. Also, of course logging will slow things down a touch itself so turn it off when you are done.
Phil
Posted: Tue Dec 06, 2005 10:08 am
by jameswilson
ok i will give that a go, im not getting any mysql processes take large amount of proc thought nbut i do have the system taking about 18% whatever that means
Posted: Tue Dec 06, 2005 3:51 pm
by jameswilson
still struggling here
Is there a way to see whats cauding the high queue ie whats in it a snapshot in time
I only have webmin access to this box mind
Posted: Tue Dec 06, 2005 11:11 pm
by jameswilson
I have tried everything i can think of and i cant get the load back to how it used to be, it used to have 30 - 30% idle time on it now its maxed
Im wondering if i have a bad stick of ram but i would expect it to fall over, i also remember enabling HT i might try disabling this unles anyone else has any suggestions
James
Posted: Tue Dec 06, 2005 11:30 pm
by victor_diago
Hi james.
Did it reached the maximum percent usage ?
It could be cleaning the disk and that surelly cause some slowdown on the server.
Victor Diago
Posted: Tue Dec 06, 2005 11:33 pm
by jameswilson
yeah but its conatantly really slow. It used to sit at about a load of 1-2 maybe peak at 3 but now its peaking at 25 and it takes 20mins (homestly) to log intot he web interface
Posted: Tue Dec 06, 2005 11:33 pm
by victor_diago
Hi james.
Did it reached the maximum percent usage ?
It could be cleaning the disk and that surelly cause some slowdown on the server.
Victor Diago
Posted: Tue Dec 06, 2005 11:44 pm
by victor_diago
Yeah, i have read again the things you said.
PLease see how many events do you have... i had problems dealing with a lot of events just right you said ... console takes a huge time to appear, thtas because mysql have to count all entries many times. that cause a lot of cpu utilization and the delay to open the console.
You could disable some queries in the console to make it faster.
Another thing that could cause cpu utilization is network monitoring, mainly if you are watching by high fps.
Wish this helps
Victor Diago
Posted: Wed Dec 07, 2005 1:14 pm
by zoneminder
If you can bear it, you could try deleting a lot (or even all) of your events and seeing if that makes a difference. That will at least give some indication of whether it is DB related.
Also try turning off zmaudit and zmfilter manually, do
and the same for zmaudit.pl
Phil