buffer overrun problems

maciekc · Post by **maciekc** » Fri Mar 31, 2006 12:24 pm

Hi

I have a problem with buffer overruns in syslog, every 15 minutes zmaudit.pl checks the consistency, it use a lot of CPU, and there are many buffer overruns, how can I prevent these overruns?
I've already set ring buffer to 50, but it didn't help a lot. Will it make diffrence when I set it to higher value(f.e. 200)?

Will it make diffrence if I change some of these options?
ZM_MAX_RESTART_DELAY
ZM_WATCH_CHECK_INTERVAL
ZM_WATCH_MAX_DELAY
ZM_OPT_FRAME_SERVER
ZM_FRAME_SOCKET_SIZE
ZM_OPT_ADAPTIVE_SKIP

I'm running P4 2.8GHz HT, 1GB RAM, SATA disk - 18 local cameras at 5fps.
When there is buffer overrun many frames are lost - about 1 minute.

Am I right that when zmc reaches the overrun the it dies?

I thought about running zmaudit only at night, but it will be problem with cleaning old events and filters. Filter will delete too much events, if the filesystem reach full. The filter will be deleting events from db, but they will be deleted only at night, so the filter will probably delete too much of events.

Do you have any ideas how to solve this problem?
Do you have the same situation in your systems?

I hope that you'll help

Best regards,
maciekc

maciekc · Post by **maciekc** » Sat Apr 01, 2006 10:39 am

I tried with
ZM_OPT_FRAME_SERVER - turned on
ZM_FRAME_SOCKET_SIZE - I set the default in kernel to 16MB
ZM_MAX_RESTART_DELAY=600
ZM_WATCH_CHECK_INTERVAL=5
ZM_WATCH_MAX_DELAY=2
ZM_OPT_ADAPTIVE_SKIP - off

I have also reniced xserver, and xlib_shm to 19 and mysql to 10 because they eat a lot of cpu. Espacially mysql when zmaudit is doing its job.

I didn't changed the ring buffer because I don't have so much memory to spend.
Did I count the space used by one monitor good? Could you explain?
352*288*3*(ring_buffer=50)/1024=14850KB - almost 14.5MB
So 18 monitors will eat 14.5*18=261MB?

here is my "free":
total used free shared buffers cached
Mem: 906000 897692 8308 0 5928 485944
-/+ buffers/cache: 405820 500180
Swap: 979924 38948 940976
As you see the system is using some swapspace, but a lot of memory is cached, can I make the buffers bigger, and lose this cache? Will it make an improvment or will make it worse?
Or maybe I should buy another 1GB or 2GB memory, and make ring buffer around 200?
Or maybe I can tweak mysql? Its settings are default,
Do you have some hints?
I really don't know what to do.

Maybe the harddisk is to slow? What do you think?

I see that when the buffer overrun is reached the system stops recording for about one minute - I have f.e. an event with frames 12:31:23-12:31:35 and the last frame is 12:32:20. So the time beetwen 12:31:35-12:32:20 is completly lost - I don't have this frames on my harddrive.

Maybe it is better to run the cleaning script(this one posted some long time ago in user contributions) in night instead of removing records by zmfilter constantly?

ps. The frames are only lost when zmaudit starts to clean the filesystem - mysql uses around 60%CPU in that time.

Please help.

victor_diago · Post by **victor_diago** » Sat Apr 01, 2006 12:39 pm

Hi,

You have just said you are using a sata disk.

Well, sata disks on linux actually sux.

try hdparm -tT /dev/sda and tell us how fast hdparm report.

I had similar problems with a damn sata that did only 20mb/s

Victor Diago

maciekc · Post by **maciekc** » Sat Apr 01, 2006 1:22 pm

Thanks for answer Victor.

here is my disk benchmark

hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 2564 MB in 2.00 seconds = 1281.55 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
Timing buffered disk reads: 138 MB in 3.05 seconds = 45.24 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

It makes 45MB so I think that's not so bad. What do you think? I think ide disks would probably do the same...

Maybe I should change hard drive to scsi with raid?

ps.
Victor did you get xlib_shm to run on zm-1.21.4?

victor_diago · Post by **victor_diago** » Sat Apr 01, 2006 1:34 pm

Hi again.

My IDE System now does 67MB/S.

Sata even is NOT a good idea yet.

I Dont know a lot about, but there are 2 main drivers for sata hardware. and both doesnt work so perfectly.

Ive issued that when using SATA disk, not only speed rate get slower, but when you are making A LOT of reads/writes to the disk, the performance of the overall system cracks...

For example, while zmaudit is running you could see this.

Ive had a problem with slackware too. As default, slackware starts a updatedb after a few minutes to reindex the database of files/locations. no problem at all right ? WRONG.

I had a few million frames (read FILES). and then the updatedb issued that starts from time to time, vanish all my stability, making the system slower.

I had A LOT of this kind of problems a few versions ago.

I really recommend you to change your disk. also you could try to enable the zmf daemon (frame server that takes the function of write the frame to the disk).

actually i have a system with 20 cameras running pretty well with 1.22.0.

Could you change your disks ?

Victor Diago

maciekc · Post by **maciekc** » Wed Apr 05, 2006 6:49 pm

victor_diago wrote:Hi again.

My IDE System now does 67MB/S.

Sata even is NOT a good idea yet.

I Dont know a lot about, but there are 2 main drivers for sata hardware. and both doesnt work so perfectly.

Ive issued that when using SATA disk, not only speed rate get slower, but when you are making A LOT of reads/writes to the disk, the performance of the overall system cracks...

For example, while zmaudit is running you could see this.

Ive had a problem with slackware too. As default, slackware starts a updatedb after a few minutes to reindex the database of files/locations. no problem at all right ? WRONG.

I had a few million frames (read FILES). and then the updatedb issued that starts from time to time, vanish all my stability, making the system slower.

I had A LOT of this kind of problems a few versions ago.

I really recommend you to change your disk. also you could try to enable the zmf daemon (frame server that takes the function of write the frame to the disk).

actually i have a system with 20 cameras running pretty well with 1.22.0.

Could you change your disks ?

Victor Diago

It is not so simple for me to change the disks. I thought of giving mysql server on the other harddrive. But probably it is not worth effort. Now I set zmaudit to run every hour. There are less lost frames, but they still exist.
Maybe I will try also to update kernel, and tune the harddisk.
I will also take a look on my mysql configuration.
If nothing will help I will use your cleaning script and run it at night when there is no movement on cameras - at night in this building there is also a traditional security system with PIR sensors - so recording in night is not so important.

Thanks for suggestions. My next system will be probably using SCSI or normal IDE.