ZoneMinder Forums

Posted: **Wed Jun 19, 2024 2:15 pm**

Hello,

Zoneminder 1.37.61
Ubuntu 20.04.6 LTS (focal)

I've been having a "floating point exception" error filling my /var/log/zm filesystem logs. The error repeats about every 1/2 millisecond and creates log files that are over 7Gb per day. Strangely, zoneminder seems to be working even when these errors are active. See the repeating error below from zm_m2.log.

Code: Select all

06/19/24 00:01:48.727562 zmc_m2[143175].ERR-zm_signal.cpp/50 [Got signal 8 (Floating point exception), crashing]
06/19/24 00:01:48.727601 zmc_m2[143175].ERR-zm_signal.cpp/82 [Signal address is 0x56147017ac47, from 0x56147017ac47]
06/19/24 00:01:48.727856 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100ac9) [0x5614701f7ac9]]
06/19/24 00:01:48.727870 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6ecbeaa420]]
06/19/24 00:01:48.727880 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0x83c47) [0x56147017ac47]]
06/19/24 00:01:48.727889 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1cc4b1) [0x5614702c34b1]]
06/19/24 00:01:48.727898 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f6ecb562df4]]
06/19/24 00:01:48.727906 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6ecbe9e609]]
06/19/24 00:01:48.727915 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6ecb24d353]]
06/19/24 00:01:48.727923 zmc_m2[143175].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
06/19/24 00:01:48.727932 zmc_m2[143175].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
06/19/24 00:01:48.727940 zmc_m2[143175].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100ac9 0x83c47 0x1cc4b1]
06/19/24 00:01:48.727954 zmc_m2[143175].ERR-zm_signal.cpp/50 [Got signal 8 (Floating point exception), crashing]
06/19/24 00:01:48.727963 zmc_m2[143175].ERR-zm_signal.cpp/82 [Signal address is 0x56147017ac47, from 0x56147017ac47]
06/19/24 00:01:48.728098 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100ac9) [0x5614701f7ac9]]
06/19/24 00:01:48.728109 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6ecbeaa420]]
06/19/24 00:01:48.728117 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0x83c47) [0x56147017ac47]]
06/19/24 00:01:48.728126 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1cc4b1) [0x5614702c34b1]]
06/19/24 00:01:48.728135 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f6ecb562df4]]
06/19/24 00:01:48.728144 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6ecbe9e609]]
06/19/24 00:01:48.728152 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6ecb24d353]]
06/19/24 00:01:48.728161 zmc_m2[143175].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
06/19/24 00:01:48.728169 zmc_m2[143175].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
06/19/24 00:01:48.728177 zmc_m2[143175].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100ac9 0x83c47 0x1cc4b1]
06/19/24 00:01:48.728188 zmc_m2[143175].ERR-zm_signal.cpp/50 [Got signal 8 (Floating point exception), crashing]

I've tried everything I could find over the past week with exhaustive googling. The last thing I did was to completely uninstall zoneminder, drop the zm table from the database, reinstall zoneminder and reconfigure my cameras from scratch. It worked for about a day, but now the error is back. Any help would be most appreciated!

Here is the output from "addr2line -Cfip -e /usr/bin/zmc 0x100ac9 0x83c47 0x1cc4b1"

Code: Select all

void std::vector<char, std::allocator<char> >::_M_realloc_insert<char>(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, char&&) at ??:?
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at ??:?
soap_getfault(soap*) at ??:?

Thanks,

Doug

Posted: **Wed Jun 19, 2024 2:24 pm**

Do you have onvif event listener on? That's the only thing that uses soap... so you could turn that off ...

We need to narrow down what it was doing before the crash. Turn on debug logs

Posted: **Wed Jun 19, 2024 2:37 pm**

Hi iconner,

Thanks for the quick reply!

"Debug" set for LOG_LEVEL_FILE, LOG_LEVEL_WEBLOG, and LOG_LEVEL_DATABASE.

ONVIF_Event_Listener: Looked at the setting for all 4 cameras and it was disabled.

Based on other posts, I checked the "DecodingEnabled" setting...

Code: Select all

select id,DecodingEnabled from Monitors;
+----+-----------------+
| id | DecodingEnabled |
+----+-----------------+
|  1 |               1 |
|  2 |               1 |
|  3 |               1 |
|  4 |               1 |
+----+-----------------+

Posted: **Wed Jun 19, 2024 2:38 pm**

Just noticed that LOG_DEBUG_LEVEL is set to 1. Should I change it to a higher number?

Posted: **Wed Jun 19, 2024 2:59 pm**

1 should be enough. 3 might be better but will be very verbose

Posted: **Wed Jun 19, 2024 3:27 pm**

Just changed LOG_DEBUG_LEVEL to 3.

My /var/log filesystem quickly filled to 100%. What files would be helpful for you to see?

/var/log/syslog: 12Gb
/var/log/zm/zmc_m2.log: 8.8Gb
/var/log/zm/zmc_m1.log: 1.7Gb

Posted: **Wed Jun 19, 2024 6:29 pm**

zmc_m?.log for whatever monitor is having the float point exception

Posted: **Wed Jun 19, 2024 7:05 pm**

I zipped them up and the file is 576Mb. It looks like the attachment limit is 1Mb. How can I get it to you?

Posted: **Wed Jun 19, 2024 7:14 pm**

Just cut out like 100 lines before the exception.

Posted: **Wed Jun 19, 2024 8:25 pm**

I've attached portions of both zmc_m1.log and zmc_m2.log. Interesting that the errors are now "Segmentation Faults" and not "Floating Point Exceptions".

zmc_m1.log.clipped.txt shows the lead-up to the repeating "Segmentation Fault" error in that file.
zmc_m2.log.clipped.txt show the lead-up to the repeating "Segmentation Fault" error in that file

zmc_m1.log.clippedAligned.txt shows the errors in the m1 file around the time that the errors in m2 file begin.

Posted: **Wed Jun 19, 2024 9:05 pm**

I tried to go into zoneminder (via web interface) to turn off the logging. I can't get the web interface to respond at all.

Would setting "ZM_WEB_LOG_DEBUG" to 0 (in the zm database config table) turn off debug logging?

Posted: **Thu Jun 20, 2024 1:10 pm**

So there's a lot of noise in there.... clearly an infinite recursion with db logging. Please set database logging to INF. I have push a fix to prevent that.
Please try again and hopefully we will get some better logs.

Posted: **Thu Jun 20, 2024 2:24 pm**

I am now:
Setting LOG_LEVEL_DATABASE to "Info"
Setting LOG_DEBUG to "checked" (via website)

I will upload log file clips as soon as I notice the error has returned.

I'm surprised that zoneminder was running this morning, because I thought I had killed and disabled it last night. Right now, it is working and the errors haven't started in the log files yet. Here's what I did last night because it wasn't responding:
1) Listed all zm processes using "ps -ef|grep zm" and killed them one by one (kill -9 [processID])
2) sudo systemctl disable zoneminder
3) I used mysql to change zm-->Config-->ZM_LOG_DEBUG to 0
4) Deleted all files in /var/log/zm
5) Rebooted the system

Posted: **Thu Jun 20, 2024 10:21 pm**

Attached are clips from zmc_m1.log and zmc_m2.log showing the Floating Point Exceptions and Segmentation fault errors.

Posted: **Fri Jun 21, 2024 1:17 am**

Have pushed a potential fix for segfault, please update and try again.

Interestingly this isn't a floating point error anymore but a straight up segfault.

ZoneMinder Forums

Floating Point Exception Filling Logs

Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs

Re: Floating Point Exception Filling Logs