Floating Point Exception Filling Logs

[dwoolridge]

Hello,

Zoneminder 1.37.61
Ubuntu 20.04.6 LTS (focal)

I've been having a "floating point exception" error filling my /var/log/zm filesystem logs. The error repeats about every 1/2 millisecond and creates log files that are over 7Gb per day. Strangely, zoneminder seems to be working even when these errors are active. See the repeating error below from zm_m2.log.

Code: Select all

06/19/24 00:01:48.727562 zmc_m2[143175].ERR-zm_signal.cpp/50 [Got signal 8 (Floating point exception), crashing]
06/19/24 00:01:48.727601 zmc_m2[143175].ERR-zm_signal.cpp/82 [Signal address is 0x56147017ac47, from 0x56147017ac47]
06/19/24 00:01:48.727856 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100ac9) [0x5614701f7ac9]]
06/19/24 00:01:48.727870 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6ecbeaa420]]
06/19/24 00:01:48.727880 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0x83c47) [0x56147017ac47]]
06/19/24 00:01:48.727889 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1cc4b1) [0x5614702c34b1]]
06/19/24 00:01:48.727898 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f6ecb562df4]]
06/19/24 00:01:48.727906 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6ecbe9e609]]
06/19/24 00:01:48.727915 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6ecb24d353]]
06/19/24 00:01:48.727923 zmc_m2[143175].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
06/19/24 00:01:48.727932 zmc_m2[143175].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
06/19/24 00:01:48.727940 zmc_m2[143175].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100ac9 0x83c47 0x1cc4b1]
06/19/24 00:01:48.727954 zmc_m2[143175].ERR-zm_signal.cpp/50 [Got signal 8 (Floating point exception), crashing]
06/19/24 00:01:48.727963 zmc_m2[143175].ERR-zm_signal.cpp/82 [Signal address is 0x56147017ac47, from 0x56147017ac47]
06/19/24 00:01:48.728098 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100ac9) [0x5614701f7ac9]]
06/19/24 00:01:48.728109 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6ecbeaa420]]
06/19/24 00:01:48.728117 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0x83c47) [0x56147017ac47]]
06/19/24 00:01:48.728126 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1cc4b1) [0x5614702c34b1]]
06/19/24 00:01:48.728135 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f6ecb562df4]]
06/19/24 00:01:48.728144 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f6ecbe9e609]]
06/19/24 00:01:48.728152 zmc_m2[143175].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6ecb24d353]]
06/19/24 00:01:48.728161 zmc_m2[143175].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
06/19/24 00:01:48.728169 zmc_m2[143175].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
06/19/24 00:01:48.728177 zmc_m2[143175].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100ac9 0x83c47 0x1cc4b1]
06/19/24 00:01:48.728188 zmc_m2[143175].ERR-zm_signal.cpp/50 [Got signal 8 (Floating point exception), crashing]

I've tried everything I could find over the past week with exhaustive googling. The last thing I did was to completely uninstall zoneminder, drop the zm table from the database, reinstall zoneminder and reconfigure my cameras from scratch. It worked for about a day, but now the error is back. Any help would be most appreciated!

Here is the output from "addr2line -Cfip -e /usr/bin/zmc 0x100ac9 0x83c47 0x1cc4b1"

Code: Select all

void std::vector<char, std::allocator<char> >::_M_realloc_insert<char>(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, char&&) at ??:?
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at ??:?
soap_getfault(soap*) at ??:?

Thanks,

Doug

[iconnor]

Do you have onvif event listener on? That's the only thing that uses soap... so you could turn that off ...

We need to narrow down what it was doing before the crash. Turn on debug logs

[dwoolridge]

Hi iconner,

Thanks for the quick reply!

"Debug" set for LOG_LEVEL_FILE, LOG_LEVEL_WEBLOG, and LOG_LEVEL_DATABASE.

ONVIF_Event_Listener: Looked at the setting for all 4 cameras and it was disabled.

Based on other posts, I checked the "DecodingEnabled" setting...

Code: Select all

select id,DecodingEnabled from Monitors;
+----+-----------------+
| id | DecodingEnabled |
+----+-----------------+
|  1 |               1 |
|  2 |               1 |
|  3 |               1 |
|  4 |               1 |
+----+-----------------+

[dwoolridge]

Just noticed that LOG_DEBUG_LEVEL is set to 1. Should I change it to a higher number?

[iconnor]

1 should be enough. 3 might be better but will be very verbose

[dwoolridge]

Just changed LOG_DEBUG_LEVEL to 3.

My /var/log filesystem quickly filled to 100%. What files would be helpful for you to see?

/var/log/syslog: 12Gb
/var/log/zm/zmc_m2.log: 8.8Gb
/var/log/zm/zmc_m1.log: 1.7Gb

[iconnor]

zmc_m?.log for whatever monitor is having the float point exception

[dwoolridge]

I zipped them up and the file is 576Mb. It looks like the attachment limit is 1Mb. How can I get it to you?

[iconnor]

Just cut out like 100 lines before the exception.

[dwoolridge]

I've attached portions of both zmc_m1.log and zmc_m2.log. Interesting that the errors are now "Segmentation Faults" and not "Floating Point Exceptions".

zmc_m1.log.clipped.txt shows the lead-up to the repeating "Segmentation Fault" error in that file.
zmc_m2.log.clipped.txt show the lead-up to the repeating "Segmentation Fault" error in that file

zmc_m1.log.clippedAligned.txt shows the errors in the m1 file around the time that the errors in m2 file begin.

[dwoolridge]

I tried to go into zoneminder (via web interface) to turn off the logging. I can't get the web interface to respond at all.

Would setting "ZM_WEB_LOG_DEBUG" to 0 (in the zm database config table) turn off debug logging?

[iconnor]

So there's a lot of noise in there.... clearly an infinite recursion with db logging. Please set database logging to INF. I have push a fix to prevent that.
Please try again and hopefully we will get some better logs.

[dwoolridge]

I am now:
Setting LOG_LEVEL_DATABASE to "Info"
Setting LOG_DEBUG to "checked" (via website)

I will upload log file clips as soon as I notice the error has returned.

I'm surprised that zoneminder was running this morning, because I thought I had killed and disabled it last night. Right now, it is working and the errors haven't started in the log files yet. Here's what I did last night because it wasn't responding:
1) Listed all zm processes using "ps -ef|grep zm" and killed them one by one (kill -9 [processID])
2) sudo systemctl disable zoneminder
3) I used mysql to change zm-->Config-->ZM_LOG_DEBUG to 0
4) Deleted all files in /var/log/zm
5) Rebooted the system

[dwoolridge]

Attached are clips from zmc_m1.log and zmc_m2.log showing the Floating Point Exceptions and Segmentation fault errors.

[iconnor]

Have pushed a potential fix for segfault, please update and try again.

Interestingly this isn't a floating point error anymore but a straight up segfault.