ZMC crashes ,memmory problems

[cefruby]

My server has been running for approx 2 years, about a month ago I upgraded to 1.2.9 (ubuntu 14) with no issues. Today I noticed individual cameras crashing....

Here is what I know:
/run/shm is pegged
lots of zma & zmc crashing
I have tried to increase /dev/shm but it does not change the situation. Things are still very unhappy.

Any ideas?
I have 5 cameras and not much has changed in the last few weeks on them at all.


df -h
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 395M 600K 394M 1% /run
/dev/sda1 913G 471G 397G 55% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 2.0G 0 100% /run/shm
none 100M 0 100M 0% /run/user
/home/cef/.Private 913G 471G 397G 55% /home/cef


syslog:
Apr 12 17:11:58 cef3 zma[4136]: ERR [Shared data not initialised by capture daemon for monitor Side-doors-high]
Apr 12 17:11:58 cef3 zmdc[4108]: ERR ['zma -m 1' exited abnormally, exit status 255]
Apr 12 17:11:58 cef3 zmdc[4108]: ERR ['zmc -m 2' exited abnormally, exit status 255]
Apr 12 17:11:58 cef3 zma[4147]: ERR [Shared data not initialised by capture daemon for monitor Driveway]
Apr 12 17:11:58 cef3 zmdc[4108]: ERR ['zma -m 2' exited abnormally, exit status 255]
Apr 12 17:11:58 cef3 X?ʰ[4149]: 9]: ERR [Got empty memory map file size 0, is the zmc process for this monitor running?]
Apr 12 17:11:58 cef3 zmdc[4108]: ERR ['zmc -m 3' exited abnormally, exit status 255]
Apr 12 17:11:59 cef3 X?ʰ[4149]: 9]: ERR [Got signal 11 (Segmentation fault), crashing]
Apr 12 17:11:59 cef3 X?ʰ[4149]: 9]: ERR [Signal address is (nil), from 0xb21f43bd]
Apr 12 17:11:59 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 0: /usr/lib/zoneminder/cgi-bin/nph-zms(_Z14zm_die_handleriP9siginfo_tPv+0x7a) [0xb2269e9a]]
Apr 12 17:11:59 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 1: [0xb218b410]]
Apr 12 17:11:59 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 2: /usr/lib/zoneminder/cgi-bin/nph-zms(_ZNK5Image10EncodeJpegEPhPii+0x8d) [0xb21f43bd]]
Apr 12 17:11:59 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 3: /usr/lib/zoneminder/cgi-bin/nph-zms(_ZN7Monitor11SingleImageEi+0xcf) [0xb220953f]]
Apr 12 17:12:00 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 4: /usr/lib/zoneminder/cgi-bin/nph-zms(_ZN13MonitorStream9runStreamEv+0xbe5) [0xb220a1e5]]
Apr 12 17:12:00 cef3 zma[4162]: ERR [Shared data not initialised by capture daemon for monitor Side-doors-high]
Apr 12 17:12:00 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 5: /usr/lib/zoneminder/cgi-bin/nph-zms(main+0xb12) [0xb21c3d12]]
Apr 12 17:12:00 cef3 zmdc[4108]: ERR ['zma -m 1' exited abnormally, exit status 255]
Apr 12 17:12:00 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 6: /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0xb0b1ea83]]
Apr 12 17:12:00 cef3 X?ʰ[4149]: 9]: ERR [Backtrace 7: /usr/lib/zoneminder/cgi-bin/nph-zms(+0x16d61) [0xb21c4d61]]
Apr 12 17:12:01 cef3 zmdc[4108]: ERR ['zmc -m 4' exited abnormally, exit status 255]
Apr 12 17:12:05 cef3 zma[4246]: ERR [Shared data not initialised by capture daemon for monitor Side-doors-high]
Apr 12 17:12:05 cef3 zmdc[4108]: ERR ['zma -m 1' exited abnormally, exit status 255]
Apr 12 17:12:07 cef3 zmfilter[4198]: ERR [No message address defined, not sending message]
Apr 12 17:12:15 cef3 zma[4288]: ERR [Shared data not initialised by capture daemon for monitor Side-doors-high]
Apr 12 17:12:15 cef3 zmdc[4108]: ERR ['zma -m 1' exited abnormally, exit status 255]
Apr 12 17:12:35 cef3 zma[4314]: ERR [Shared data not initialised by capture daemon for monitor Side-doors-high]
Apr 12 17:12:35 cef3 zmdc[4108]: ERR ['zma -m 1' exited abnormally, exit status 255]
cef@cef3:~$

[bbunge]

I had a similar problem with 1.28.107. There was a database fix.
Did you upgrade the database?

[knight-of-ni]

Like you mentioned, this right here is the problem:

Code: Select all

none 2.0G 2.0G 0 100% /run/shm

This is what zoneminder uses for shared memory so its no wonder why things aren't working.
You said you tried to increase it, but you did not say how or by how much.

Stop zoneminder, and see how much frees up. If the amount free is not in the 90-100% range then you either have stale files in there or you've got something else using /run/shm.

[cefruby]

i increased it to 8gig still getting mem alloc errors. The SWAP is not clearing out even with ZM ,apache,mysql stopped . But I cannot determine what is using it and hogging all the machines memmory. I am not running anything else on this machine.


seeing this now:
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 395M 604K 394M 1% /run
/dev/sda1 913G 465G 402G 54% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 8.0G 3.3G 4.8G 42% /run/shm
none 100M 0 100M 0% /run/user
/home/cef/.Private 913G 465G 402G 54% /home/cef

and getting this:
Apr 12 18:14:14 cef3 zmdc[1690]: INF [Starting pending process, zma -m 7]
Apr 12 18:14:15 cef3 zma[1840]: FAT [Memory allocation failed: Cannot allocate memory]
Apr 12 18:14:16 cef3 zmdc[1690]: FAT [Can't fork: Cannot allocate memory]
Apr 12 18:14:16 cef3 zmc[1819]: FAT [Unable to decode frame at frame 0]

[cefruby]

this is interesting. despite turning off the zoneminder ,apache and mysql service .the following were still running and eating all that mem. any thoughts on how to fix that? (it's repeatable after reboot as well)


1842 www-data 20 0 960680 18520 116 S 0.0 0.5 0:00.12 zma
1770 www-data 20 0 25012 7504 0 S 0.0 0.2 0:00.08 zmupdate.pl
1760 www-data 20 0 24044 6416 28 S 0.0 0.2 0:00.07 zmaudit.pl

[cefruby]

I did upgrade DB as part of 1.2.9 upgrade.

[knight-of-ni]

Sure, with zoneminder stopped, kill the remaining zm* processes from the command line.

BTW: the version is 1.29.0, not 1.2.9.

[cefruby]

thank you knnigget. I have killed the remaining services .but the memmory apparently does not free up . 30 min later
cef@cef3:~$ free -m
total used free shared buffers cached
Mem: 3942 3538 404 3361 21 3441
-/+ buffers/cache: 75 3867

cef@cef3:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 2.0G 4.0K 2.0G 1% /dev
tmpfs 395M 596K 394M 1% /run
/dev/sda1 913G 465G 402G 54% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 8.0G 3.3G 4.8G 42% /run/shm
none 100M 0 100M 0% /run/user
/home/cef/.Private 913G 465G 402G 54% /home/cef
cef@cef3:~$

[bbunge]

Stop mysql

[cefruby]

I could not free up the memmory and found some post where someone claimed a kernel upgrade was the way to go. Long story short: upgraded to ubuntu 16 (via apt-get) and found nasty surprises like mysql 5.7. It did fix my memmory issues but the cameras were still crashing alot.
I built a new server this morning on a different machine (haswell i3) fresh ubuntu 15 & 1.29. It too was crashing cameras (signal 134) even when only set to recrord mode (so no motion detection). I found that very odd but started tuning capture frame rate and capture size until i got the machine to behave. I later replicated this on my original i5. I now have my 5 cameras wih modect working reliably again but recording less than I want (and did for over 2 years) .


not sure what changed ,but guessing its something to do with 1.29

[bbunge]

I've gotten Ubuntu 16.04 final beta working with MySQL or Mariadb. Preliminary instructions on the WIKI.

If you are having problems with Ubuntu may I recommend you try CentOS 7 with ZM 1.29.0. I recommend you use an XFS file system without LVM,

bb

[knight-of-ni]

Wait a minute. This does not make sense.

cef@cef3:~$ free -m
total used free shared buffers cached
Mem: 3942 3538 404 3361 21 3441
-/+ buffers/cache: 75 3867

And

none 8.0G 3.3G 4.8G 42% /run/shm

Did you really set /dev/shm to 8G when you only have 4G of physical ram in your system!?!?!????
Out of memory is out of memory. There is no magical software setting, which will make more memory appear in your system.

If this is really the case, then the possible solutions are to a) reconfigure your monitors to use less memory or b) add more physical memory.
If you want to do the former, then we need to see a complete monitor config for all your cameras.

[cefruby]

Yeh, I did that and did not keep it. Below is the current running config (my big memmory not freeing problem seems resolved).

I am still seeing lots of buffer overruns and crashing. I have 5 cameras all ffmpeg pulling approx 5 fps. Is there a simple way to print out the running monitor configs? I'd like to fix this without adding memmory , since I think this is ought to be sufficient.

one other note . I am seeing cameras that are set to 'record' go into alarm state -should that be possible?!



cef@cef3:~$ cef@cef3:~$ cef@cef3:~$ total used free shared buff/cache available
Mem: 3945 1611 130 1599 2203 691
Swap: 0 0 0
cef@cef3:~$ df: 0h: No such file or directory
cef@cef3:~$ Filesystem Size Used Avail Use% Mounted on
udev 2.0G 0 2.0G 0% /dev
tmpfs 395M 6.0M 389M 2% /run
/dev/sda1 913G 338G 529G 39% /
tmpfs 2.0G 1.6G 388M 81% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 395M 0 395M 0% /run/user/1000
/home/cef/.Private 913G 338G 529G 39% /home/cef

[knight-of-ni]

You'll either need to show screenshots of each General, Source, and Buffers tab or write the field/value pairs down.
Naturally, if these cameras are identical, you only need to show one set, otherwise all 5.

[cefruby]

here goes:


Camera FRONT (2)
server : none
Source Type: FFMPEG
function: Mocord
enabled : yes
NO LINKED MONITORS
Analysis FPS 5.00
Maximum FPS (?) 15.00
Alarm Maximum FPS (?) 30.00
Reference Image Blend: 12.5
AlmRefImageBlendPct: 6.25%
Triggers None available
Source Path
rtsp://XXX:XXXX@192.168.0.81:554/live2.sdp
Remote Method : RTP/RTSP
Options (?) :none
Target colorspace 32bit
Capture Width (pixels): 1920
Capture Height (pixels): 1080
Preserve Aspect Ratio
Orientation Normal
Deinterlacing disabled
Image Buffer Size (frames) 50
Warmup Frames 25
Pre Event Image Count 25
Post Event Image Count 25
Stream Replay Image Buffer 1000
Alarm Frame Count 1

Camera Driveway (3)
server : none
Source Type: FFMPEG
function: Mocord
enabled : yes
NO LINKED MONITORS
Analysis FPS 5.00
Maximum FPS (?) 6.00
Alarm Maximum FPS (?) 30.00
Reference Image Blend: 12.5
AlmRefImageBlendPct: 6.25%
Triggers None available
Source Path
rtsp://XXX:XXXX@192.168.0.84:554/live3.sdp
Remote Method : RTP/RTSP
Options (?) :none
Target colorspace 32bit
Capture Width (pixels): 1920
Capture Height (pixels): 1080
Preserve Aspect Ratio
Orientation Normal
Deinterlacing disabled
Image Buffer Size (frames) 50
Warmup Frames 25
Pre Event Image Count 25
Post Event Image Count 25
Stream Replay Image Buffer 1000
Alarm Frame Count 1

Back (4)
server : none
Source Type: FFMPEG
function: Mocord
enabled : yes
NO LINKED MONITORS
Analysis FPS 0.00
Maximum FPS (?) 6.00
Alarm Maximum FPS (?) 30.00
Reference Image Blend: 12.5
AlmRefImageBlendPct: 6.25%
Triggers None available
Source Path
rtsp://XXX:XXXX@192.168.0.82:554/live2.sdp
Remote Method : RTP/RTSP
Options (?) :none
Target colorspace 32bit
Capture Width (pixels): 1360
Capture Height (pixels): 768
Preserve Aspect Ratio
Orientation Normal
Deinterlacing disabled
Image Buffer Size (frames) 50
Warmup Frames 25
Pre Event Image Count 25
Post Event Image Count 25
Stream Replay Image Buffer 1000
Alarm Frame Count 1

sidedoors (5)
server : none
Source Type: FFMPEG
function: Mocord
enabled : yes
NO LINKED MONITORS
Analysis FPS 3.00
Maximum FPS (?) 3.0
Alarm Maximum FPS (?) 6.00
Reference Image Blend: 12.5 %
AlmRefImageBlendPct: 6.25%
Triggers None available
Source Path
rtsp://XXX:XXXX@192.168.0.80:554/live3.sdp
Remote Method : RTP/RTSP
Options (?) :none
Target colorspace 32bit
Capture Width (pixels): 1360
Capture Height (pixels): 768
Preserve Aspect Ratio
Orientation Normal
Deinterlacing disabled
Image Buffer Size (frames) 50
Warmup Frames 25
Pre Event Image Count 25
Post Event Image Count 25
Stream Replay Image Buffer 1000
Alarm Frame Count 1

Hikvision (7)
server : none
Source Type: FFMPEG
function: Mocord
enabled : yes
NO LINKED MONITORS
Analysis FPS 2.00
Maximum FPS (?) 5.00
Alarm Maximum FPS (?) 16.00
Reference Image Blend: 12.5
AlmRefImageBlendPct: 6.25%
Triggers None available
Source Path
rtsp://xxx:xxxx@192.168.0.85/streaming/Channels/1Remote Method : RTP/RTSP
Options (?) :none
Target colorspace 32bit
Capture Width (pixels): 1920
Capture Height (pixels): 1080
Preserve Aspect Ratio
Orientation Normal
Deinterlacing disabled
Image Buffer Size (frames) 50
Warmup Frames 25
Pre Event Image Count 25
Post Event Image Count 25
Stream Replay Image Buffer 1000
Alarm Frame Count 1