After upgrade to 1.29, crashing daily

Forum for questions and support relating to the 1.29.x releases only.
Forum rules
Locked
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

After upgrade to 1.29, crashing daily

Post by Basildane »

I have a system that was stable for years, and I upgraded to 1.29.0 recently via apt-get.
This is Ubuntu server 14.04. CPU is i5-3570K. I think it's 16 gigs of RAM.
Since upgrading it has been crashing regularly. The main web page displays "Stopped", but the channels continue to run and record events.

I tried to export the log, but that function returns "error 500", so here is a copy/paste of some recent log entries.
I will gladly post whatever else might be helpful.

Code: Select all

2016-03-03 19:03:46.728860	zmdc		12218	ERR	'zma -m 1' exited abnormally, exit status 11	zmdc.pl	
2016-03-03 19:03:45.997557	zma_m1		12977	ERR	Backtrace 8: /usr/bin/zma(+0x1f1f7) [0x7f128bbda1f7]	zm_signal.cpp	105
2016-03-03 19:03:45.839038	zma_m1		12977	ERR	Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f128808dec5]	zm_signal.cpp	105
2016-03-03 19:03:45.722330	zma_m1		12977	ERR	Backtrace 6: /usr/bin/zma(main+0x3d4) [0x7f128bbd9944]	zm_signal.cpp	105
2016-03-03 19:03:45.567076	zma_m1		12977	ERR	Backtrace 5: /usr/bin/zma(_ZN7MonitorD1Ev+0x1ad) [0x7f128bc1863d]	zm_signal.cpp	105
2016-03-03 19:03:45.017906	zma_m1		12977	ERR	Backtrace 4: /usr/bin/zma(_ZN12LibvlcCameraD0Ev+0x9) [0x7f128bcb9f79]	zm_signal.cpp	105
2016-03-03 19:03:44.753957	zma_m1		12977	ERR	Backtrace 3: /usr/bin/zma(_ZN12LibvlcCameraD1Ev+0x85) [0x7f128bcb9d65]	zm_signal.cpp	105
2016-03-03 19:03:44.603792	zma_m1		12977	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f12880eee0c]	zm_signal.cpp	105
2016-03-03 19:03:44.503581	zma_m1		12977	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f128aad4340]	zm_signal.cpp	105
2016-03-03 19:03:44.353534	zma_m1		12977	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7f128bc71642]	zm_signal.cpp	105
2016-03-03 19:03:44.253210	zma_m1		12977	ERR	Signal address is (nil), from 0x7f12880eee0c	zm_signal.cpp	84
2016-03-03 19:03:44.053151	zma_m1		12977	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 19:03:43.200650	zmdc		12218	ERR	'zma -m 9' exited abnormally, exit status 11	zmdc.pl	
2016-03-03 19:03:42.740636	zma_m9		12431	ERR	Backtrace 8: /usr/bin/zma(+0x1f1f7) [0x7fb5282eb1f7]	zm_signal.cpp	105
2016-03-03 19:03:42.640189	zma_m9		12431	ERR	Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb52479eec5]	zm_signal.cpp	105
2016-03-03 19:03:42.539869	zma_m9		12431	ERR	Backtrace 6: /usr/bin/zma(main+0x3d4) [0x7fb5282ea944]	zm_signal.cpp	105
2016-03-03 19:03:42.422935	zma_m9		12431	ERR	Backtrace 5: /usr/bin/zma(_ZN7MonitorD1Ev+0x1ad) [0x7fb52832963d]	zm_signal.cpp	105
2016-03-03 19:03:42.172565	zma_m9		12431	ERR	Backtrace 4: /usr/bin/zma(_ZN12LibvlcCameraD0Ev+0x9) [0x7fb5283caf79]	zm_signal.cpp	105
2016-03-03 19:03:42.005442	zma_m9		12431	ERR	Backtrace 3: /usr/bin/zma(_ZN12LibvlcCameraD1Ev+0x85) [0x7fb5283cad65]	zm_signal.cpp	105
2016-03-03 19:03:41.680023	zma_m9		12431	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7fb5247ffe0c]	zm_signal.cpp	105
2016-03-03 19:03:41.446563	zma_m9		12431	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fb5271e5340]	zm_signal.cpp	105
2016-03-03 19:03:41.195433	zma_m9		12431	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7fb528382642]	zm_signal.cpp	105
2016-03-03 19:03:41.095200	zma_m9		12431	ERR	Signal address is (nil), from 0x7fb5247ffe0c	zm_signal.cpp	84
2016-03-03 19:03:40.944638	zma_m9		12431	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 19:01:12.887857	web_js		12054	ERR	Export request failed: 500 / Internal Server Error - exportFail()	?view=log	
2016-03-03 19:01:08.713956	web_js		12054	ERR	Export request failed: 500 / Internal Server Error - exportFail()	?view=log	
2016-03-03 18:59:59.737154	web_js		13244	ERR	Export request failed: 500 / Internal Server Error - exportFail()	?view=log	
2016-03-03 18:49:14.772167	zms		13251	ERR	Unable to send stream frame: Broken pipe	zm_monitor.cpp	4127
2016-03-03 18:48:19.365410	web_php		13194	ERR	Timed out waiting for msg /var/run/zm/zms-204509s.sock	includes/functions.php	2371
2016-03-03 18:48:16.466133	zms		13199	ERR	Terminating, last frame sent time 1457048895.966012 secs more than maximum of 10.000000	zm_monitor.cpp	4392
2016-03-03 17:53:06.384420	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:52:25.378040	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:47:23.836130	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:45:12.774190	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:41:30.776990	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:33:48.947650	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:30:07.768870	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:29:57.226440	zmwatch		12407	ERR	Error getting last capture time for Polling-House	zmwatch.pl	
2016-03-03 17:23:13.857420	zmdc		12218	ERR	'zma -m 1' exited abnormally, exit status 11	zmdc.pl	
2016-03-03 17:23:13.462233	zma_m1		12957	ERR	Backtrace 8: /usr/bin/zma(+0x1f1f7) [0x7f2f180ef1f7]	zm_signal.cpp	105
2016-03-03 17:23:13.361832	zma_m1		12957	ERR	Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f2f145a2ec5]	zm_signal.cpp	105
2016-03-03 17:23:13.261392	zma_m1		12957	ERR	Backtrace 6: /usr/bin/zma(main+0x3d4) [0x7f2f180ee944]	zm_signal.cpp	105
2016-03-03 17:23:13.161328	zma_m1		12957	ERR	Backtrace 5: /usr/bin/zma(_ZN7MonitorD1Ev+0x1ad) [0x7f2f1812d63d]	zm_signal.cpp	105
2016-03-03 17:23:13.043890	zma_m1		12957	ERR	Backtrace 4: /usr/bin/zma(_ZN12LibvlcCameraD0Ev+0x9) [0x7f2f181cef79]	zm_signal.cpp	105
2016-03-03 17:23:12.885407	zma_m1		12957	ERR	Backtrace 3: /usr/bin/zma(_ZN12LibvlcCameraD1Ev+0x85) [0x7f2f181ced65]	zm_signal.cpp	105
2016-03-03 17:23:12.785151	zma_m1		12957	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f2f14603e0c]	zm_signal.cpp	105
2016-03-03 17:23:12.684978	zma_m1		12957	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f2f16fe9340]	zm_signal.cpp	105
2016-03-03 17:23:12.584849	zma_m1		12957	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7f2f18186642]	zm_signal.cpp	105
2016-03-03 17:23:12.476230	zma_m1		12957	ERR	Signal address is (nil), from 0x7f2f14603e0c	zm_signal.cpp	84
2016-03-03 17:23:12.376228	zma_m1		12957	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 17:11:00.197450	zmdc		12218	ERR	'zma -m 1' exited abnormally, exit status 11	zmdc.pl	
2016-03-03 17:10:59.923115	zma_m1		12433	ERR	Backtrace 8: /usr/bin/zma(+0x1f1f7) [0x7ff1ed0bc1f7]	zm_signal.cpp	105
2016-03-03 17:10:59.822727	zma_m1		12433	ERR	Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7ff1e956fec5]	zm_signal.cpp	105
2016-03-03 17:10:59.772446	zma_m1		12433	ERR	Backtrace 6: /usr/bin/zma(main+0x3d4) [0x7ff1ed0bb944]	zm_signal.cpp	105
2016-03-03 17:10:59.722209	zma_m1		12433	ERR	Backtrace 5: /usr/bin/zma(_ZN7MonitorD1Ev+0x1ad) [0x7ff1ed0fa63d]	zm_signal.cpp	105
2016-03-03 17:10:59.663672	zma_m1		12433	ERR	Backtrace 4: /usr/bin/zma(_ZN12LibvlcCameraD0Ev+0x9) [0x7ff1ed19bf79]	zm_signal.cpp	105
2016-03-03 17:10:59.605207	zma_m1		12433	ERR	Backtrace 3: /usr/bin/zma(_ZN12LibvlcCameraD1Ev+0x85) [0x7ff1ed19bd65]	zm_signal.cpp	105
2016-03-03 17:10:59.553965	zma_m1		12433	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7ff1e95d0e0c]	zm_signal.cpp	105
2016-03-03 17:10:59.503733	zma_m1		12433	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7ff1ebfb6340]	zm_signal.cpp	105
2016-03-03 17:10:59.453673	zma_m1		12433	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7ff1ed153642]	zm_signal.cpp	105
2016-03-03 17:10:59.403364	zma_m1		12433	ERR	Signal address is (nil), from 0x7ff1e95d0e0c	zm_signal.cpp	84
2016-03-03 17:10:59.353196	zma_m1		12433	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 15:11:09.182890	zmwatch		12407	ERR	Error getting last capture time for Upper-Yard	zmwatch.pl	
2016-03-03 15:11:08.350130	zmwatch		12407	ERR	Error getting last capture time for Front-Door	zmwatch.pl	
2016-03-03 15:10:48.865560	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:10:47.639080	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:10:36.486120	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:10:25.721440	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:10:14.901670	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:10:04.263960	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:10:01.401390	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:10:01.351321	zma_m10		12636	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:09:56.341810	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:09:56.240799	zma_m7		12634	ERR	Shared data not initialised by capture daemon for monitor Front-Door	zm_monitor.cpp	442
2016-03-03 15:09:53.574690	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:09:43.003110	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:09:32.391150	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:09:21.699900	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:09:10.352410	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:59.585870	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:48.890050	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:41.367890	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:08:41.314180	zma_m10		12563	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:08:38.281980	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:36.319910	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:08:36.270218	zma_m7		12551	ERR	Shared data not initialised by capture daemon for monitor Front-Door	zm_monitor.cpp	442
2016-03-03 15:08:27.510310	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:16.665200	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:05.856950	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:08:01.503400	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:08:01.385783	zma_m10		12522	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:56.481650	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:56.381315	zma_m7		12520	ERR	Shared data not initialised by capture daemon for monitor Front-Door	zm_monitor.cpp	442
2016-03-03 15:07:56.288040	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:55.231530	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.10' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:07:54.741070	zmwatch		12407	ERR	Memory map file '/dev/shm/zm.mmap.7' should have been 896 but was instead 0	zmwatch.pl	
2016-03-03 15:07:52.494080	zmdc		12218	ERR	'zma -m 6' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:52.393452	zma_m6		12502	ERR	Shared data not initialised by capture daemon for monitor Front-Yard	zm_monitor.cpp	442
2016-03-03 15:07:50.204149	zms		12498	ERR	Frame send time 1752 msec too slow, throttling maxfps to 3.33	zm_monitor.cpp	4140
2016-03-03 15:07:41.836210	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:41.735544	zma_m10		12496	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:39.637260	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:36.354950	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:36.302164	zma_m7		12491	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:32.609640	zmdc		12218	ERR	'zma -m 6' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:32.509066	zma_m6		12459	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:31.392270	zmdc		12218	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:31.308484	zma_m5		12448	ERR	Shared data not initialised by capture daemon for monitor Upper-Yard-North	zm_monitor.cpp	442
2016-03-03 15:07:31.208020	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:31.057967	zma_m10		12443	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:26.873320	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:26.689294	zma_m7		12437	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:25.570480	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:25.362239	zma_m10		12427	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:25.178410	zmdc		12218	ERR	'zma -m 11' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:24.836820	zma_m11		12425	ERR	Shared data not initialised by capture daemon for monitor Polling-House	zm_monitor.cpp	442
2016-03-03 15:07:23.343040	zmdc		12218	ERR	'zma -m 11' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:23.126587	zma_m11		12391	ERR	Shared data not initialised by capture daemon for monitor Polling-House	zm_monitor.cpp	442
2016-03-03 15:07:23.051100	zmdc		12218	ERR	'zmc -m 11' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:22.942710	zmdc		12218	ERR	'zma -m 10' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:22.592725	zma_m10		12377	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:22.125440	zmdc		12218	ERR	'zma -m 6' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:21.875569	zma_m6		12372	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:21.541610	zmdc		12218	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:21.308379	zma_m5		12366	ERR	Shared data not initialised by capture daemon for monitor Upper-Yard-North	zm_monitor.cpp	442
2016-03-03 15:07:20.749340	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:20.432617	zma_m7		12359	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:19.881900	zmdc		12218	ERR	'zma -m 9' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:19.482270	zma_m9		12355	ERR	Shared data not initialised by capture daemon for monitor Driveway	zm_monitor.cpp	442
2016-03-03 15:07:19.139810	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:18.123060	zmdc		12218	ERR	'zma -m 9' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:17.980821	zma_m9		12343	ERR	Shared data not initialised by capture daemon for monitor Driveway	zm_monitor.cpp	442
2016-03-03 15:07:17.788680	zmdc		12218	ERR	'zmc -m 9' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:17.554150	zmdc		12218	ERR	'zma -m 8' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:17.312510	zma_m8		12333	ERR	Shared data not initialised by capture daemon for monitor Barn	zm_monitor.cpp	442
2016-03-03 15:07:17.312230	zmdc		12218	ERR	'zmc -m 8' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:17.153800	zmdc		12218	ERR	'zma -m 7' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:16.870502	zma_m7		12323	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:16.870240	zmdc		12218	ERR	'zma -m 6' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:16.679012	zma_m6		12321	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:16.419570	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:16.320410	zmdc		12218	ERR	'zma -m 6' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:16.219523	zma_m6		12311	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:15.835320	zmdc		12218	ERR	'zma -m 3' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:15.685589	zma_m3		12302	ERR	Shared data not initialised by capture daemon for monitor Garden	zm_monitor.cpp	442
2016-03-03 15:07:15.584990	zmdc		12218	ERR	'zma -m 1' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:15.435518	zma_m1		12300	ERR	Shared data not initialised by capture daemon for monitor Princes-Lane	zm_monitor.cpp	442
2016-03-03 15:07:15.434900	zmdc		12218	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:15.284828	zma_m5		12298	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:14.883970	zmdc		12218	ERR	'zma -m 4' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:14.833890	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:14.734199	zma_m4		12294	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:14.282970	zmdc		12218	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:14.232910	zmdc		12218	ERR	'zma -m 4' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:14.182810	zmdc		12218	ERR	'zma -m 3' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:14.132730	zmdc		12218	ERR	'zma -m 2' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:14.032610	zmdc		12218	ERR	'zmc -d /dev/video0' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:13.984154	zma_m5		12287	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:13.982610	zmdc		12218	ERR	'zma -m 1' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:13.884080	zmdc		12218	ERR	'zmc -m 1' exited abnormally, exit status 255	zmdc.pl	
2016-03-03 15:07:13.882954	zma_m2		12254	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:13.882830	zma_m3		12261	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:13.882731	zma_m4		12268	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:13.832594	zma_m1		12246	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	533
2016-03-03 15:07:10.623594	zma_m8		1662	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f8ff890fe0c]	zm_signal.cpp	105
2016-03-03 15:07:10.623164	zma_m1		11629	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f356c515e0c]	zm_signal.cpp	105
2016-03-03 15:07:10.572997	zma_m9		11307	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7fefd82ede0c]	zm_signal.cpp	105
2016-03-03 15:07:10.523513	zma_m8		1662	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f8ffb2f5340]	zm_signal.cpp	105
2016-03-03 15:07:10.523062	zma_m1		11629	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f356eefb340]	zm_signal.cpp	105
2016-03-03 15:07:10.472981	zma_m9		11307	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fefdacd3340]	zm_signal.cpp	105
2016-03-03 15:07:10.424203	zma_m8		1662	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7f8ffc492642]	zm_signal.cpp	105
2016-03-03 15:07:10.423325	zma_m1		11629	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7f3570098642]	zm_signal.cpp	105
2016-03-03 15:07:10.373151	zma_m9		11307	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7fefdbe70642]	zm_signal.cpp	105
2016-03-03 15:07:10.077440	zma_m1		11629	ERR	Signal address is (nil), from 0x7f356c515e0c	zm_signal.cpp	84
2016-03-03 15:07:10.076931	zma_m8		1662	ERR	Signal address is (nil), from 0x7f8ff890fe0c	zm_signal.cpp	84
2016-03-03 15:07:09.922581	zma_m9		11307	ERR	Signal address is (nil), from 0x7fefd82ede0c	zm_signal.cpp	84
2016-03-03 15:07:09.781800	zma_m1		11629	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 15:07:09.781497	zma_m8		1662	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 15:07:09.780824	zma_m9		11307	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 09:20:55.262830	zmwatch		1684	ERR	Unable to run "/usr/bin/zmdc.pl restart zmc -m 8", output is "Unable to connect to server"	zmwatch.pl	
2016-03-03 09:11:37.413770	zmdc		1391	ERR	'zma -m 1' exited abnormally, exit status 11	zmdc.pl	
2016-03-03 09:11:37.048521	zma_m1		11597	ERR	Backtrace 8: /usr/bin/zma(+0x1f1f7) [0x7fc576dd51f7]	zm_signal.cpp	105
2016-03-03 09:11:36.906624	zma_m1		11597	ERR	Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fc573288ec5]	zm_signal.cpp	105
2016-03-03 09:11:36.806460	zma_m1		11597	ERR	Backtrace 6: /usr/bin/zma(main+0x3d4) [0x7fc576dd4944]	zm_signal.cpp	105
2016-03-03 09:11:36.672858	zma_m1		11597	ERR	Backtrace 5: /usr/bin/zma(_ZN7MonitorD1Ev+0x1ad) [0x7fc576e1363d]	zm_signal.cpp	105
2016-03-03 09:11:36.572816	zma_m1		11597	ERR	Backtrace 4: /usr/bin/zma(_ZN12LibvlcCameraD0Ev+0x9) [0x7fc576eb4f79]	zm_signal.cpp	105
2016-03-03 09:11:36.430933	zma_m1		11597	ERR	Backtrace 3: /usr/bin/zma(_ZN12LibvlcCameraD1Ev+0x85) [0x7fc576eb4d65]	zm_signal.cpp	105
2016-03-03 09:11:36.330740	zma_m1		11597	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7fc5732e9e0c]	zm_signal.cpp	105
2016-03-03 09:11:36.171173	zma_m1		11597	ERR	Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fc575ccf340]	zm_signal.cpp	105
2016-03-03 09:11:36.054639	zma_m1		11597	ERR	Backtrace 0: /usr/bin/zma(_Z14zm_die_handleriP9siginfo_tPv+0x72) [0x7fc576e6c642]	zm_signal.cpp	105
2016-03-03 09:11:35.895795	zma_m1		11597	ERR	Signal address is (nil), from 0x7fc5732e9e0c	zm_signal.cpp	84
2016-03-03 09:11:35.787504	zma_m1		11597	ERR	Got signal 11 (Segmentation fault), crashing	zm_signal.cpp	54
2016-03-03 09:03:23.066280	zmdc		1391	ERR	'zma -m 1' exited abnormally, exit status 11	zmdc.pl	
2016-03-03 09:03:22.644535	zma_m1		11526	ERR	Backtrace 8: /usr/bin/zma(+0x1f1f7) [0x7f7c9ec0f1f7]	zm_signal.cpp	105
2016-03-03 09:03:22.544361	zma_m1		11526	ERR	Backtrace 7: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7c9b0c2ec5]	zm_signal.cpp	105
2016-03-03 09:03:22.443775	zma_m1		11526	ERR	Backtrace 6: /usr/bin/zma(main+0x3d4) [0x7f7c9ec0e944]	zm_signal.cpp	105
2016-03-03 09:03:22.335074	zma_m1		11526	ERR	Backtrace 5: /usr/bin/zma(_ZN7MonitorD1Ev+0x1ad) [0x7f7c9ec4d63d]	zm_signal.cpp	105
2016-03-03 09:03:22.176402	zma_m1		11526	ERR	Backtrace 4: /usr/bin/zma(_ZN12LibvlcCameraD0Ev+0x9) [0x7f7c9eceef79]	zm_signal.cpp	105
2016-03-03 09:03:22.067739	zma_m1		11526	ERR	Backtrace 3: /usr/bin/zma(_ZN12LibvlcCameraD1Ev+0x85) [0x7f7c9eceed65]	zm_signal.cpp	105
2016-03-03 09:03:21.909048	zma_m1		11526	ERR	Backtrace 2: /lib/x86_64-linux-gnu/libc.so.6(cfree+0x1c) [0x7f7c9b123e0c]	zm_signal.cpp	105
Top
bbunge
Posts: 2956
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: After upgrade to 1.29, crashing daily

Post by bbunge »

Sorry for your problem. There are some things you need to do after the upgrade.

See: https://wiki.zoneminder.com/Common_Issu ... PA_version
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

Thanks bbunge. Yes, I followed that guide when I upgraded, to the letter.
Server runs for hours, then randomly quits.
Top
bbunge
Posts: 2956
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: After upgrade to 1.29, crashing daily

Post by bbunge »

Looks like you are running out of space in the tmpfs /dev/shm With 16 GIG of RAM Ubuntu would grab about 8 GIG for tmpfs by default. That means that you likely have lots of cameras with the camera resolution set high or there is something else using space in /dev/shm. A "df" will tell you how much of the /dev/shm is being used. If it is running close you can add an entry to fstab to increase the size of /dev/shm Be aware, though, how much memory MySQL needs. Install and run mysqltuner. I'm running 17 cameras at low resolution with almost 70K events in 30 days and I've had to set my innodb_buffer_pool_size to 2G to keep the data in memory. With my low resolution my system only uses about 5% of 6 GIG tmpfs (12 GIG total RAM).
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

Interesting. I have 10 cameras online now, 4 of them are high-def. This is a requirement. In fact, they are not high enough def!
This system has run for years, I think starting around 1.24 or earlier.

This is my df output.

Code: Select all

Filesystem     1K-blocks      Used Available Use% Mounted on
udev             8150000         4   8149996   1% /dev
tmpfs            1632160       688   1631472   1% /run
/dev/sda2      944407980 752717872 143693848  84% /
none                   4         0         4   0% /sys/fs/cgroup
none                5120         0      5120   0% /run/lock
none             8160784   7194516    966268  89% /run/shm
none              102400         0    102400   0% /run/user
/dev/sda1         497696      3456    494240   1% /boot/efi
root@zoneminder:/dev/shm#

root@zoneminder:/dev/shm# df -H /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
none            8.4G  7.4G  990M  89% /run/shm
Top
rip
Posts: 38
Joined: Sun Sep 02, 2012 8:59 pm

Re: After upgrade to 1.29, crashing daily

Post by rip »

Code: Select all

none             8160784   7194516    966268  89% /run/shm
That's 89% full, too close for my personal comfort.

The last line in (the stock ubuntu distro) /etc/fstab gives an example of how to change this.

Also for reference:
https://wiki.zoneminder.com/Math_for_Me ... o_optimize
For some discussion on the topic.
Top
bbunge
Posts: 2956
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: After upgrade to 1.29, crashing daily

Post by bbunge »

I agree with rip. You should make more room in /run/shm (/dev/shm is in later versions of Ubuntu). As I said, though, don't rob too much and cause issues with MySQL...

bb
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

Well cool, I learned something new about linux. I'll do some work on that and see how it goes. Thanks.
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

I got my tempfs down to 49%, but it is still crashing.

Code: Select all

root@zoneminder:/dev# df -H /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            12G  5.7G  6.1G  49% /run/shm
Top
rip
Posts: 38
Joined: Sun Sep 02, 2012 8:59 pm

Re: After upgrade to 1.29, crashing daily

Post by rip »

Are you still getting the same segfault and /dev/shm errors in your log?

Wondering if a lot of triggers/alarms are also running up the shm usage, or otherwise maybe have to run memtest against your system and see if there is a memory fault?

You could also do a

Code: Select all

watch -n 1 df -H /dev/shm
in a terminal session and trigger alarm on a bunch of your cameras and see if you are over running the available space.

Running top in yet another session would let you see if per chance you are doing something silly at the same time utilization wise as well.
Top
SteveGilvarry
Posts: 494
Joined: Sun Jun 29, 2014 1:12 pm
Location: Melbourne, AU

Re: After upgrade to 1.29, crashing daily

Post by SteveGilvarry »

I think I saw m1 and m9 having segmentation faults, are they similar cameras and ZM settings? Maybe just try a camera at a time by setting others to None just to see if these cameras are faulting on their own or if it is the combined load of all the cameras running. Also did I see libvlc in the seg fault? Maybe try same camera on ffmpeg. Not saying there isn't a problem in 1.29, I fixed a segmentation fault in libvlc that was stopping monitors loading just before 1.29, so could be other issues for sure.
Production Zoneminder 1.37.x (Living dangerously)
Random Selection of Cameras (Dahua and Hikvision)
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

> SteveGilvarry
Yes, m1 and m9 are GXV3672-HD cameras.

> rip
Interesting, I will try that.

I'm getting this error every 5 seconds. I don't know what it means. All the cameras are up and streaming.

Code: Select all

2016-03-10 14:29:47.890516	web_js		2403	ERR	getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-316786s.sock ) failed: No such file or directory - checkStreamForErrors()	?view=watch	
2016-03-10 14:29:47.830257	web_php		2403	ERR	socket_sendto( /var/run/zm/zms-316786s.sock ) failed: No such file or directory	includes/functions.php	2371
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

Oh, I forgot to post, I ran Memtest86 this weekend and found no errors.
Top
User avatar
Basildane
Posts: 108
Joined: Mon Sep 16, 2013 1:09 pm

Re: After upgrade to 1.29, crashing daily

Post by Basildane »

I have not made any config changes since last weekend.
Checked server today, and 90% of recorded events were 0 length.

So I'm back to running memtest86 again. Completely down now.

/dev/shm is consistently 50%.
Top
User avatar
gnocera
Posts: 34
Joined: Thu Jun 04, 2015 6:31 pm

Re: After upgrade to 1.29, crashing daily

Post by gnocera »

We are having trouble with this too.

2 server, both with Ubuntu 14.04 lts, both with zoneminder 1.29.0, both stopping with similar socket errors.
Each server has 16-cores (2 x CPU AMD Opteron 4000 8-core ), 16GB RAM, 8TB disk and around 70 HD H264 cameras.

We didn't have those problems with zoneminder 1.28. Problem just started with ZM 1.29.0
We decided to upgrade because we did some test with 1.29 and we realized that 1.29 save some resources from the CPU, so CPU run more smoother.
But unfortunately Zoneminder stops suddenly.

Did you fix your problem BASILADANE or still in troubles?

Thank you.
Top
Locked

Return to “ZoneMinder 1.29.x”