ZoneMinder Forums

I have a largish (21 cams) zm server that stopped,
The last event in the log was out of memory.
When i got to the machine i could change tty (alt ctrl f1 2 etc) type in a user name but thats all it would do.

I can post the full log i have removed the machine and will pwer it up again later

James

I had this exact same thing happen to a RedHat 9.0 box that ZM wasn't installed on - Out of Memory displayed on the screen a bunch of times, won't log in on a different tty, etc. I've got an IM server, several web servers, anti-virus update server, masquerading, mysql server, data extraction system, etc, etc, etc (basically if I need a server for something, this is the machine I go to) loaded on it with only 256 meg of RAM. While I don't know what caused it, like you I reset it. It's been fine ever since, although I'm going to shove more RAM into it in the near future. (The system is fairly mission-critical though, so I couldn't have it down any more than absolutely necessary.)

Sorry if this is OT with nothing to do with ZM, but it seemed too similar to not mention.

i have had this again
relevant messages log below

Oct 24 23:26:22 localhost zmc_d1[2042]: WAR [Buffer overrun at index 35 ]
Oct 24 23:26:26 localhost kernel: oom-killer: gfp_mask=0x80d2
Oct 24 23:26:26 localhost kernel: DMA per-cpu:
Oct 24 23:26:26 localhost kernel: cpu 0 hot: low 2, high 6, batch 1
Oct 24 23:26:26 localhost kernel: cpu 0 cold: low 0, high 2, batch 1
Oct 24 23:26:26 localhost kernel: Normal per-cpu:
Oct 24 23:26:26 localhost kernel: cpu 0 hot: low 62, high 186, batch 31
Oct 24 23:26:26 localhost kernel: cpu 0 cold: low 0, high 62, batch 31
Oct 24 23:26:26 localhost kernel: HighMem per-cpu:
Oct 24 23:26:26 localhost kernel: cpu 0 hot: low 62, high 186, batch 31
Oct 24 23:26:26 localhost kernel: cpu 0 cold: low 0, high 62, batch 31
Oct 24 23:26:26 localhost kernel:
Oct 24 23:26:27 localhost zmc_d0[2909]: WAR [Buffer overrun at index 11 ]
Oct 24 23:26:27 localhost kernel: Free pages: 39952kB (496kB HighMem)
Oct 24 23:26:28 localhost kernel: Active:207530 inactive:211635 dirty:0 writeback:0 unstable:0 free:9988 slab:3328 mapped:350500 pagetables:1452
Oct 24 23:26:28 localhost kernel: DMA free:7168kB min:68kB low:84kB high:100kB active:3104kB inactive:2684kB present:16384kB pages_scanned:84742 all_unreclaimable? no
Oct 24 23:26:28 localhost kernel: lowmem_reserve[]: 0 880 1775
Oct 24 23:26:28 localhost kernel: Normal free:32288kB min:3756kB low:4692kB high:5632kB active:377640kB inactive:387832kB present:901120kB pages_scanned:4392195 all_unreclaimable? no
Oct 24 23:26:28 localhost kernel: lowmem_reserve[]: 0 0 7167
Oct 24 23:26:28 localhost kernel: HighMem free:496kB min:512kB low:640kB high:768kB active:449376kB inactive:456024kB present:917440kB pages_scanned:2224073 all_unreclaimable? no
Oct 24 23:26:28 localhost kernel: lowmem_reserve[]: 0 0 0
Oct 24 23:26:28 localhost kernel: DMA: 0*4kB 0*8kB 0*16kB 66*32kB 21*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 7168kB
Oct 24 23:26:28 localhost kernel: Normal: 32*4kB 0*8kB 1084*16kB 319*32kB 2*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 32288kB
Oct 24 23:26:28 localhost kernel: HighMem: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 496kB
Oct 24 23:26:28 localhost kernel: Swap cache: add 148083, delete 148000, find 4712266/4714078, race 0+8
Oct 24 23:26:28 localhost kernel: Free swap = 0kB
Oct 24 23:26:28 localhost kernel: Total swap = 530104kB
Oct 24 23:26:28 localhost kernel: Out of Memory: Killed process 963 (zmc).
Oct 24 23:26:28 localhost zma_m10[2166]: INF [Cam_9: 801063 - Gone back into alarm state]
Oct 24 23:26:28 localhost zma_m10[2166]: INF [Cam_9: 801064 - Gone into alert state]
Oct 24 23:26:29 localhost zma_m7[1778]: INF [Cam_6: 799048 - Left alarm state (32664) - 259(22) images]
Oct 24 23:26:29 localhost zma_m9[1954]: INF [Cam_8: 799032 - Gone back into alarm state]
Oct 24 23:26:29 localhost zma_m9[1954]: INF [Cam_8: 799033 - Gone into alert state]
Oct 24 23:26:29 localhost zma_m16[2758]: INF [Cam_15: 769840 - Gone into alert state]
Oct 24 23:26:29 localhost zma_m16[2758]: INF [Cam_15: 769841 - Gone back into alarm state]
Oct 24 23:26:31 localhost zma_m12[2352]: INF [Cam_11: 801192 - Gone into alarm state]

Hi jamesWilson.

I have had this a lot of times in 16 channels setup, what i can say is that this is the normal thing that system does.

When the operating system have much low memory (or it get full at all) it uses swap... when both, true memory and swap get full this is what you get, a out of memory with some process get killed.


In my cases, i set swap always to 512MB and memory for 16 channels to 1GB. I really recommend you to put 2GB of ram and 512MB swap, you can even set it to lower fps that reduces memory usage, or turn some cameras in RECORD instead of MODECT.

Memory is very cheap, even here in Brazil that the USdollar are 3 times more expensive, i really think that buy two memory sticks of 1 GB each is the best that can be done.

Wish you lucky !

Victor Diago

Thanks victor i agree with the 2gig thing but i have 2 gig of ram in it
top reports 1.3 gig unused
I also have a 512Mb swap.
I dont mind it running out of memory but the last time it did this (with 1 gig) it actually fell over. This time it recovered but the system stopped recording for approx 7 mins
I suppose this was the time while it was dying to when it recovered. i dont mind it running out of mem but i do mind it being down for 1 min much less 7
Im assuming its a error or bug somewhere or can i make it restart the process before i get this out of memory issue?

I have checked a couple of smaller systems and they dont have this error or any kernel warnings in the logs, this larger system has a feww out of memory errors logged?

I would say that it appears to have choked out from having no free swap.
What distro, and kernel?

Okay, i agree, look at this :

this really seems to be that the out of memory situation was caused by no free swap space.

BUT i think that if you already have 2 Gb of ram, the swap should not be so much used.


at what fps are you capturing ?

Victor Diago

Some input for you - my systems all have a gig of ram and 2 gig of swap space. (Except for that general all-purpose server mentioned above of course - 256 meg RAM and 512 meg swap in that one.) I figured that hard drive space is cheap, so why not throw as much at it as it can use? I remember that I was planning on giving it 5 gig for swapping, but ran into some problem where it would only take 2; don't remember the specifics though.

I've got one system running 24 cameras recording 24x7 at 320x240 at 2.5 fps and another running 8 cameras recording 24x7 at 640x480 at 1 fps. None of my zoneminder systems have had this problem set up as above with slackware. (The general server is running redhat 9, at least until I have time to rebuild it - it's days are numbered. )

For some reason the swap doesn't get cleared like it should. I have noticed this with Fedora. I don't know if this is a kernel issue maybe?
What is responcible for handling swap?

I know the 2.6 kernel still has some issues so might this be related?
I automaticly turn off acpi since I know that the 2.6 kern has issues with it.

So what kernel is every one else that has this problem using?
What kernel is your slack Ruler?
I know that RH9 was 2.4 Kernel so this might be the case maybe.

Regards,
Cordel

The RH9 box that had this problem once that I am aware of is running kernel 2.4.20-8

The older of the ZM boxes is running slackware 10.0 and kernel 2.4.26-29 (custom built with DVD+RW Tools and v4l2 kraxel patches applied).

The newer of my ZM boxes is running slackware 10.1 and kernel 2.4.29-4 (also custom built with the above patches).

well i first had it on my first post with a 2.6.11 kernel, when i had the out of mem issue i updated everything (not zm related) and the kernel to 2.6.12. I agree that hd is cheap but i assumed that give it enough ram it wont need to swap. I have always used .5 gig of swap but would appear not to be enough on this one. I have had a look at the system via webmin and it says it has

Real memory: 1816368 kB total / 1094748 kB free Swap space: 530104 kB total / 220644 kB free

Now it has 4 sticks of 512 kingston in it i assume the rest of the ram is hidden as something internal?

it shows 2 gig in bios at start

what i dont understand is why its using any swap when it has a gig of spare ram?

I dont have this mem issue on any other machines using this distro on this or the older kernel.

BTW i am capturing all cams 16 pal, and 5 axis 206 at 2ish fps

the axis are 640x480 and the videos are 368x288 most at 24 bit
all in mocord on default zone settings. 4 x 250 gig striped raid 0 var

Hi Cordel.

Can you please explain what are the implications of disabling acpi, and how to do it?

Thanks.

First question is this for a new install or existing?
Second thing is unless you are getting errors related to you avanced power management I would leave it alone.

Regards,
Cordel

Hi Cordel.

It's for an existing install of FC4/ZM. No errors so far, but it would be nice to know it in ahead, if I would encounter such "out of memory" problems.

jameswilson: Set swap size to 2xRAMsize