Floating Point Exception Filling Logs

[MarsRover]

Another thing that might be important:
It all started after I did apt-get full-upgrade on this system 2 days ago, which also upgraded ZM.

To get rid of these errors I downgraded ZM consequently to:
1.37.61~20240705153736-focal
1.37.61~20240703085514-focal
1.37.61~20240628094110-focal
But neither helped, getting same errors.
So I returned to to latest 1.37.62~20240707094607-focal.

All these builds don't have issues on the other box, where I didn't run Ubuntu full-upgrade.
The only major difference I found is on that "stable" box I have apache2-* 2.4.41-4ubuntu3.17, while the faulting one has 2.4.41-4ubuntu3.20

Could this be the cause?

[iconnor]

There are some signficant changes between 1.37.60 and 1.37.62 regarding how/when we can free up the ram in packets. We weren't freeing up any while recording, even though we are likely done with the packet, so using more ram than necessary. Freeing up the ram required changes to how the event writing thread accesses the packetqueue and does locking. It introduced crashes. I have fixed several, but there is at least one more. My machines don't crash, but I see very infrequent crashes on some servers I maintain. Some people like you all get them pretty reliably.

The problem is exacerbated by the backtraces being almost useless these days.

[MarsRover]

There are some signficant changes between 1.37.60 and 1.37.62 regarding how/when we can free up the ram in packets.
...........
The problem is exacerbated by the backtraces being almost useless these days.

Thanks for the feedback!
Will downgrading to 1.37.59 or even (.57) work as a temporary workaround until this gets fixed?

[iconnor]

I may have found the problem. Certainly found and fixed A problem.
Packages building.

[MarsRover]

I may have found the problem. Certainly found and fixed A problem.
Packages building.

Great!
Meanwhile, just FYI: downgrading to latest 37.59 (1.37.59~20240508.78-focal) did the trick for me.

[iconnor]

Please take note of ram usage. I believe once we get the crashing fixed, it will use significantly less ram when recording. At least that is the hope

[dwoolridge]

Not sure if you still need them, but here are my log files with "Crashing" in them: (debug level 4)
v1.37.62

zmc_m1.log

Code: Select all

07/14/24 14:29:04.620118 zmc_m1[680838].ERR-zmc.cpp/318 [Failed to capture image from monitor 1 Back Porch (1/1)]
07/14/24 15:06:45.421444 zmc_m1[719523].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 15:06:45.421483 zmc_m1[719523].ERR-zm_signal.cpp/82 [Signal address is 0x32343a363008, from 0x55927f8cf5f1]
07/14/24 15:06:45.421742 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x55927f8feb89]]
07/14/24 15:06:45.421751 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9e0c9fe420]]
07/14/24 15:06:45.421756 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0xd15f1) [0x55927f8cf5f1]]
07/14/24 15:06:45.421761 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1eab2c) [0x55927f9e8b2c]]
07/14/24 15:06:45.421766 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f9e0c0b6df4]]
07/14/24 15:06:45.421770 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9e0c9f2609]]
07/14/24 15:06:45.421775 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f9e0bda1353]]
07/14/24 15:06:45.421780 zmc_m1[719523].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 15:06:45.421785 zmc_m1[719523].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 15:06:45.421790 zmc_m1[719523].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89 0xd15f1 0x1eab2c]
07/14/24 15:06:45.421800 zmc_m1[719523].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 15:06:45.421805 zmc_m1[719523].ERR-zm_signal.cpp/82 [Signal address is 0x32343a363008, from 0x55927f8cf5f1]
07/14/24 15:06:45.421934 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x55927f8feb89]]
07/14/24 15:06:45.421941 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9e0c9fe420]]
07/14/24 15:06:45.421945 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0xd15f1) [0x55927f8cf5f1]]
07/14/24 15:06:45.421950 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1eab2c) [0x55927f9e8b2c]]
07/14/24 15:06:45.421955 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f9e0c0b6df4]]
07/14/24 15:06:45.421960 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9e0c9f2609]]
07/14/24 15:06:45.421964 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f9e0bda1353]]
07/14/24 15:06:45.421968 zmc_m1[719523].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 15:06:45.421973 zmc_m1[719523].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 15:06:45.421977 zmc_m1[719523].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89 0xd15f1 0x1eab2c]
07/14/24 15:06:45.421985 zmc_m1[719523].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 15:06:45.421990 zmc_m1[719523].ERR-zm_signal.cpp/82 [Signal address is 0x32343a363008, from 0x55927f8cf5f1]
07/14/24 15:06:45.422129 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x55927f8feb89]]
07/14/24 15:06:45.422135 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9e0c9fe420]]
07/14/24 15:06:45.422140 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 2: /usr/bin/zmc(+0xd15f1) [0x55927f8cf5f1]]
07/14/24 15:06:45.422145 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 3: /usr/bin/zmc(+0x1eab2c) [0x55927f9e8b2c]]
07/14/24 15:06:45.422150 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 4: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7f9e0c0b6df4]]
07/14/24 15:06:45.422154 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9e0c9f2609]]
07/14/24 15:06:45.422159 zmc_m1[719523].ERR-zm_signal.cpp/108 [Backtrace 6: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f9e0bda1353]]
07/14/24 15:06:45.422163 zmc_m1[719523].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 15:06:45.422168 zmc_m1[719523].ERR-zm_signal.cpp/128 [and execute the following command for more information:]07/14/24 15:06:45.422172 zmc_m1[719523].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89 0xd15f1 0x1eab2c]
07/14/24 15:06:45.422181 zmc_m1[719523].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]

zmc_m2.log

Code: Select all

07/14/24 11:14:35.204791 zmc_m2[680843].ERR-zmc.cpp/318 [Failed to capture image from monitor 2 Garage (1/1)]
07/14/24 11:16:25.896277 zmc_m2[703224].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 11:16:25.896308 zmc_m2[703224].ERR-zm_signal.cpp/82 [Signal address is 0xc, from 0x7fb2a9d4e4ad]
07/14/24 11:16:25.906887 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x5613af7ddb89]]
07/14/24 11:16:25.906904 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb2a991c420]]
07/14/24 11:16:25.906911 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 2: /lib/x86_64-linux-gnu/libavutil.so.56(av_buffer_ref+0x3d) [0x7fb2a9d4e4ad]]
07/14/24 11:16:25.906917 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 3: /lib/x86_64-linux-gnu/libavcodec.so.58(av_packet_ref+0x38) [0x7fb2aa0c8f48]]
07/14/24 11:16:25.906923 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 4: /usr/bin/zmc(+0x22e3b0) [0x5613af90b3b0]]
07/14/24 11:16:25.906928 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 5: /usr/bin/zmc(+0x2369af) [0x5613af9139af]]
07/14/24 11:16:25.906934 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 6: /usr/bin/zmc(+0x1e84ff) [0x5613af8c54ff]]
07/14/24 11:16:25.906939 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 7: /usr/bin/zmc(+0x1eb23d) [0x5613af8c823d]]
07/14/24 11:16:25.906944 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7fb2a8fd4df4]]
07/14/24 11:16:25.906948 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb2a9910609]]
07/14/24 11:16:25.906953 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 10: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fb2a8cbf353]]
07/14/24 11:16:25.906958 zmc_m2[703224].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 11:16:25.906963 zmc_m2[703224].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 11:16:25.906968 zmc_m2[703224].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89 0x22e3b0 0x2369af 0x1e84ff 0x1eb23d]
07/14/24 11:16:25.907012 zmc_m2[703224].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 11:16:25.907017 zmc_m2[703224].ERR-zm_signal.cpp/82 [Signal address is 0xc, from 0x7fb2a9d4e4ad]
07/14/24 11:16:25.907181 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x5613af7ddb89]]
07/14/24 11:16:25.907189 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb2a991c420]]
07/14/24 11:16:25.907194 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 2: /lib/x86_64-linux-gnu/libavutil.so.56(av_buffer_ref+0x3d) [0x7fb2a9d4e4ad]]
07/14/24 11:16:25.907199 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 3: /lib/x86_64-linux-gnu/libavcodec.so.58(av_packet_ref+0x38) [0x7fb2aa0c8f48]]
07/14/24 11:16:25.907204 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 4: /usr/bin/zmc(+0x22e3b0) [0x5613af90b3b0]]
07/14/24 11:16:25.907209 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 5: /usr/bin/zmc(+0x2369af) [0x5613af9139af]]
07/14/24 11:16:25.907214 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 6: /usr/bin/zmc(+0x1e84ff) [0x5613af8c54ff]]
07/14/24 11:16:25.907219 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 7: /usr/bin/zmc(+0x1eb23d) [0x5613af8c823d]]
07/14/24 11:16:25.907224 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 8: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6df4) [0x7fb2a8fd4df4]]
07/14/24 11:16:25.907229 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 9: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb2a9910609]]
07/14/24 11:16:25.907234 zmc_m2[703224].ERR-zm_signal.cpp/108 [Backtrace 10: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fb2a8cbf353]]
07/14/24 11:16:25.907238 zmc_m2[703224].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 11:16:25.907243 zmc_m2[703224].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 11:16:25.907248 zmc_m2[703224].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89 0x22e3b0 0x2369af 0x1e84ff 0x1eb23d]
07/14/24 11:16:25.907256 zmc_m2[703224].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]

zmc_m3.log

Code: Select all

07/14/24 11:15:25.297726 zmc_m3[701614].DB3-zm_packetqueue.cpp/245 [stream index 0 ?= video_stream_id 0, keyframe 0, keep_keyframes 1,  counts 18 > pre_event_count 5 at begin 1]
07/14/24 11:15:25.297728 zmc_m3[701614].DB4-zm_packet.h/117 [packet 6038 unlocked, 0x5630981c1240, locked 1, owns 1]
07/14/24 11:15:25.297730 zmc_m3[701614].DB4-zm_packet.h/120 [packet 6038 unlocked, 0x5630981c1240, locked 0, owns 0]
07/14/24 11:15:25.297732 zmc_m3[701614].DB2-zm_packetqueue.cpp/561 [Incrementing 0x56309818d960, queue size 18, end? 0, deleting 0]
07/14/24 11:15:25.297733 zmc_m3[701614].DB2-zm_packetqueue.cpp/568 [Incrementing 0x56309818d960, 0x56309818d960 still not at end, so returning true]
07/14/24 11:15:25.297735 zmc_m3[701614].DB4-zm_packetqueue.cpp/465 [Locking in get_packet using it 0x56309818d960 queue end? 0]
07/14/24 11:15:25.297737 zmc_m3[701614].DB4-zm_packetqueue.cpp/472 [Have Lock in get_packet]
07/14/24 11:15:25.297739 zmc_m3[701614].DB3-zm_packetqueue.cpp/485 [get_packet using it 0x56309818d960 locking index 6039]
07/14/24 11:15:25.297741 zmc_m3[701614].DB4-zm_packet.h/110 [TryLocking packet 6039 0x56309817d500 locked: 0 owns: 0]
07/14/24 11:15:25.297742 zmc_m3[701614].DB4-zm_packet.h/112 [TryLocking packet 6039 0x56309817d500 1, owns: 1]
07/14/24 11:15:25.297744 zmc_m3[701614].DB2-zm_packetqueue.cpp/490 [Locked packet 6039, unlocking packetqueue mutex]
07/14/24 11:15:25.297746 zmc_m3[701614].DB3-zm_monitor.cpp/2094 [Motion detection is enabled?(2) signal(1) signal_change(0) trigger state(Cancel) image index 6039]
07/14/24 11:15:25.297748 zmc_m3[701614].DB4-zm_monitor.cpp/2107 [Trigger not OFF state is (0)]
07/14/24 11:15:25.297749 zmc_m3[701614].DB4-zm_monitor.cpp/2219 [Not linked_monitors]
07/14/24 11:15:25.297751 zmc_m3[701614].DB4-zm_monitor.h/718 [Ready because image_count(94482) >= ready_count(5)]
07/14/24 11:15:25.297753 zmc_m3[701614].DB3-zm_monitor.cpp/2236 [signal and capturing and doing motion detection 2]
07/14/24 11:15:25.297754 zmc_m3[701614].DB1-zm_monitor.cpp/2241 [Recalculating motion_frame_skip (2) = capture_fps(4.999832) / analysis_fps(2.000000)]
07/14/24 11:15:25.297757 zmc_m3[701614].DB1-zm_monitor.cpp/2303 [Skipped motion detection last motion score was 1]
07/14/24 11:15:25.297758 zmc_m3[701614].DB1-zm_monitor.cpp/2311 [Blending full colour image because analysis_image = 1, in_frame=0x7f0974665a40 and format 0 != 0, 12]
07/14/24 11:15:25.298205 zmc_m3[701614].DB1-zm_monitor.cpp/2318 [Done Blending]
07/14/24 11:15:25.298208 zmc_m3[701614].DB1-zm_monitor.cpp/2366 [!score state=ALARM, snap->score -1]
07/14/24 11:15:25.298210 zmc_m3[701614].DB1-zm_monitor.cpp/2401 [Event 58558, alarm frames 0 <? alarm_frame_count 1 duration:0close mode 4]
07/14/24 11:15:25.298229 zmc_m3[701614].DB3-zm_packetqueue.cpp/245 [stream index 0 ?= video_stream_id 0, keyframe 0, keep_keyframes 1,  counts 18 > pre_event_count 5 at begin 1]
07/14/24 11:15:25.298231 zmc_m3[701614].DB4-zm_packet.h/117 [packet 6039 unlocked, 0x56309817d500, locked 1, owns 1]
07/14/24 11:15:25.298233 zmc_m3[701614].DB4-zm_packet.h/120 [packet 6039 unlocked, 0x56309817d500, locked 0, owns 0]
07/14/24 11:15:25.298235 zmc_m3[701614].DB2-zm_packetqueue.cpp/561 [Incrementing 0x56309818d960, queue size 18, end? 0, deleting 0]
07/14/24 11:15:25.298237 zmc_m3[701614].DB2-zm_packetqueue.cpp/568 [Incrementing 0x56309818d960, 0x56309818d960 still not at end, so returning true]
07/14/24 11:15:25.298238 zmc_m3[701614].DB4-zm_packetqueue.cpp/465 [Locking in get_packet using it 0x56309818d960 queue end? 0]
07/14/24 11:15:25.298240 zmc_m3[701614].DB4-zm_packetqueue.cpp/472 [Have Lock in get_packet]
07/14/24 11:15:25.298242 zmc_m3[701614].DB3-zm_packetqueue.cpp/485 [get_packet using it 0x56309818d960 locking index 6040]
07/14/24 11:15:25.298244 zmc_m3[701614].DB4-zm_packet.h/110 [TryLocking packet 6040 0x56309817b900 locked: 0 owns: 0]
07/14/24 11:15:25.298246 zmc_m3[701614].DB4-zm_packet.h/112 [TryLocking packet 6040 0x56309817b900 1, owns: 1]
07/14/24 11:15:25.298247 zmc_m3[701614].DB2-zm_packetqueue.cpp/490 [Locked packet 6040, unlocking packetqueue mutex]
07/14/24 11:15:25.298249 zmc_m3[701614].DB3-zm_monitor.cpp/2094 [Motion detection is enabled?(2) signal(1) signal_change(0) trigger state(Cancel) image index 6040]
07/14/24 11:15:25.298251 zmc_m3[701614].DB4-zm_monitor.cpp/2107 [Trigger not OFF state is (0)]
07/14/24 11:15:25.298252 zmc_m3[701614].DB4-zm_monitor.cpp/2219 [Not linked_monitors]
07/14/24 11:15:25.298254 zmc_m3[701614].DB4-zm_monitor.h/718 [Ready because image_count(94482) >= ready_count(5)]
07/14/24 11:15:25.298256 zmc_m3[701614].DB3-zm_monitor.cpp/2236 [signal and capturing and doing motion detection 2]
07/14/24 11:15:25.298257 zmc_m3[701614].DB1-zm_monitor.cpp/2241 [Recalculating motion_frame_skip (2) = capture_fps(4.999832) / analysis_fps(2.000000)]
07/14/24 11:15:25.298260 zmc_m3[701614].DB1-zm_monitor.cpp/2261 [Detecting motion on image 6040, image 0x7f0974666760]
07/14/24 11:15:25.298676 zmc_m3[701614].DB3-zm_monitor.cpp/3253 [Blanking inactive zone Exclusion_01]
07/14/24 11:15:25.299785 zmc_m3[701614].DB3-zm_monitor.cpp/3253 [Blanking inactive zone Exclusion_02]
07/14/24 11:15:25.300125 zmc_m3[701614].DB3-zm_monitor.cpp/3301 [Checking active zone MotionArea]
07/14/24 11:15:25.300183 zmc_m3[701614].DB4-zm_zone.cpp/227 [Checking alarms for zone 4/MotionArea in lines 0 -> 719]
07/14/24 11:15:25.305409 zmc_m3[680849].DB2-zm_ffmpeg_camera.cpp/254 [ffmpeg_camera in: pts: 109281985 * 1/90000=1214.244278, dts: 109281985, size: 1373, stream_index: 0, video flags: 0000, keyframe(0) pos: -1, duration: 18000]
07/14/24 11:15:25.305425 zmc_m3[680849].DB4-zm_monitor.cpp/2759 [Back from capture result=1 image count 6043]
07/14/24 11:15:25.305429 zmc_m3[680849].DB2-zm_monitor.cpp/2788 [Have packet stream_index:0 ?= videostream_id: 0 q.vpktcount 18 event? 1 image_count 6043]
07/14/24 11:15:25.305434 zmc_m3[680849].DB2-zm_packetqueue.cpp/141 [packet counts for 0 is 19]
07/14/24 11:15:25.305438 zmc_m3[680849].DB4-zm_packetqueue.cpp/215 [packetqueue queuepacket, unlocked signalling]
07/14/24 11:15:25.305448 zmc_m3[680849].DB4-zm_ffmpeg_camera.cpp/223 [Using video input because 1214416500 >= 1214244278]
07/14/24 11:15:25.305456 zmc_m3[701613].DB3-zm_packetqueue.cpp/532 [get_packet using it 0x563098217e20 locking index 6043]
07/14/24 11:15:25.305470 zmc_m3[701613].DB4-zm_packet.h/110 [TryLocking packet 6043 0x5630981c24e0 locked: 0 owns: 0]
07/14/24 11:15:25.305475 zmc_m3[701613].DB4-zm_packet.h/112 [TryLocking packet 6043 0x5630981c24e0 1, owns: 1]
07/14/24 11:15:25.305479 zmc_m3[701613].DB2-zm_packetqueue.cpp/537 [Locked packet 6043, unlocking packetqueue mutex, incrementing it]
07/14/24 11:15:25.305512 zmc_m3[701613].DB4-zm_packet.cpp/99 [about to decode video, image_index is (6043)]
07/14/24 11:15:25.305531 zmc_m3[703169].DB3-zm_packetqueue.cpp/485 [get_packet using it 0x7f0974060fc0 locking index 5997]
07/14/24 11:15:25.305552 zmc_m3[703169].DB4-zm_packet.h/110 [TryLocking packet 5997 0x5630981a4db0 locked: 0 owns: 0]
07/14/24 11:15:25.305558 zmc_m3[703169].DB4-zm_packet.h/112 [TryLocking packet 5997 0x5630981a4db0 1, owns: 1]
07/14/24 11:15:25.305563 zmc_m3[703169].DB2-zm_packetqueue.cpp/490 [Locked packet 5997, unlocking packetqueue mutex]
07/14/24 11:15:25.305567 zmc_m3[703169].DB2-zm_packetqueue.cpp/561 [Incrementing 0x7f0974060fc0, queue size 19, end? 0, deleting 0]
07/14/24 11:15:25.305571 zmc_m3[703169].DB2-zm_packetqueue.cpp/568 [Incrementing 0x7f0974060fc0, 0x7f0974060fc0 still not at end, so returning true]
07/14/24 11:15:25.305575 zmc_m3[703169].DB1-zm_event.cpp/702 [Adding packet 5997]
07/14/24 11:15:25.305579 zmc_m3[703169].DB2-zm_event.cpp/322 [have_video_keyframe 1 codec_type 0 == video? 1 packet keyframe 0]
07/14/24 11:15:25.305583 zmc_m3[703169].DB2-zm_event.cpp/324 [Adding to event: pts: 139678510780176, dts: -9223372036854775808, size: 0, stream_index: 0, flags: 0000, keyframe(0) pos: -1, duration: 0]
07/14/24 11:15:25.305589 zmc_m3[703169].DB1-zm_videostore.cpp/1041 [Queue size for 0 is 0]
07/14/24 11:15:25.305594 zmc_m3[703169].DB1-zm_videostore.cpp/1070 [Pushing on queue 0, size is 1]
07/14/24 11:15:25.305599 zmc_m3[703169].DB2-zm_videostore.cpp/1281 [Doing passthrough, just copy packet: pts: 139678510780176 * 1/90000=1551983453.113067, dts: -9223372036854775808, size: 0, stream_index: 0, video flags: 0000, keyframe(0) pos: -1, duration: 0]
07/14/24 11:15:25.305612 zmc_m3[703169].DB1-zm_videostore.cpp/1396 [undef dts, fixing by setting to stream last_dts 21540]
07/14/24 11:15:25.305616 zmc_m3[703169].DB2-zm_videostore.cpp/1435 [finished pkt: pts: 139678402365816 * 1/90000=1551982248.509067, dts: 21540, size: 0, stream_index: 0, video flags: 0000, keyframe(0) pos: -1, duration: 0]
07/14/24 11:15:25.305622 zmc_m3[703169].DB3-zm_videostore.cpp/1436 [next_dts for stream 0 has become 39540 last_dts 21540]
07/14/24 11:15:25.305633 zmc_m3[703169].DB4-zm_videostore.cpp/1443 [Success writing packet]
07/14/24 11:15:25.305637 zmc_m3[703169].DB1-zm_event.cpp/413 [Have frame type Normal from score(0) state 3 frames 3 bulk frame interval 100 and mod3]
07/14/24 11:15:25.305641 zmc_m3[703169].DB1-zm_event.cpp/428 [frames 3, score 0 max_score 0]
07/14/24 11:15:25.305644 zmc_m3[703169].DB1-zm_event.cpp/432 [Writing snapshot to /mnt/data/zm_recording/3/2024-07-14/58558/snapshot.jpg]
07/14/24 11:15:25.305749 zmc_m3[703169].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 11:15:25.305775 zmc_m3[703169].DB1-zm_signal.cpp/57 [Signal information: number 11 code 1 errno 0 pid 0 uid 0 status 0]
07/14/24 11:15:25.305779 zmc_m3[703169].ERR-zm_signal.cpp/82 [Signal address is (nil), from 0x7f0995b472f9]
07/14/24 11:15:25.305894 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x563095215b89]]
07/14/24 11:15:25.305903 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f0995823420]]
07/14/24 11:15:25.305910 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 2: /lib/x86_64-linux-gnu/libjpeg.so.8(+0x3f2f9) [0x7f0995b472f9]]
07/14/24 11:15:25.305917 zmc_m3[703169].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 11:15:25.305923 zmc_m3[703169].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 11:15:25.305929 zmc_m3[703169].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89]
07/14/24 11:15:25.305941 zmc_m3[703169].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 11:15:25.305947 zmc_m3[703169].DB1-zm_signal.cpp/57 [Signal information: number 11 code 1 errno 0 pid 0 uid 0 status 0]
07/14/24 11:15:25.305951 zmc_m3[703169].ERR-zm_signal.cpp/82 [Signal address is (nil), from 0x7f0995b472f9]
07/14/24 11:15:25.305970 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x563095215b89]]
07/14/24 11:15:25.305985 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f0995823420]]
07/14/24 11:15:25.305992 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 2: /lib/x86_64-linux-gnu/libjpeg.so.8(+0x3f2f9) [0x7f0995b472f9]]
07/14/24 11:15:25.305998 zmc_m3[703169].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 11:15:25.306005 zmc_m3[703169].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 11:15:25.306011 zmc_m3[703169].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89]
07/14/24 11:15:25.306021 zmc_m3[703169].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]
07/14/24 11:15:25.306028 zmc_m3[703169].DB1-zm_signal.cpp/57 [Signal information: number 11 code 1 errno 0 pid 0 uid 0 status 0]
07/14/24 11:15:25.306032 zmc_m3[703169].ERR-zm_signal.cpp/82 [Signal address is (nil), from 0x7f0995b472f9]
07/14/24 11:15:25.306047 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 0: /usr/bin/zmc(+0x100b89) [0x563095215b89]]
07/14/24 11:15:25.306054 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f0995823420]]
07/14/24 11:15:25.306061 zmc_m3[703169].ERR-zm_signal.cpp/108 [Backtrace 2: /lib/x86_64-linux-gnu/libjpeg.so.8(+0x3f2f9) [0x7f0995b472f9]]
07/14/24 11:15:25.306067 zmc_m3[703169].ERR-zm_signal.cpp/127 [Backtrace complete, please install debug symbols (typically zoneminder-dbg)]
07/14/24 11:15:25.306073 zmc_m3[703169].ERR-zm_signal.cpp/128 [and execute the following command for more information:]
07/14/24 11:15:25.306080 zmc_m3[703169].ERR-zm_signal.cpp/129 [addr2line -Cfip -e /usr/bin/zmc 0x100b89]
07/14/24 11:15:25.306089 zmc_m3[703169].ERR-zm_signal.cpp/50 [Got signal 11 (Segmentation fault), crashing]

[MarsRover]

Please take note of ram usage. I believe once we get the crashing fixed, it will use significantly less ram when recording. At least that is the hope

I'm monitoring it and yes, I can say it's more efficient now. Especially when I explicitly specify MaxImageBufferCount.
Also, segfaults are rare now.

On two lates builds (1.37.62~20240713100122-focal and 1.37.63~20240715.8-focal) I periodically get tons of messages like this for some cameras in Database Log, but not in file log, while both are set to Level = WAR:

Code: Select all

ERR   Got unexpected memory map file size 37330480, expected 37330472    zm_monitor.cpp	1036

I'm getting them 2 times per second for each monitor affected.

And sometimes all of a sudden they just stop appearing for no apparent reason.

[MarsRover]

in Database Log, but not in file log, while both are set to Level = WAR:

Code: Select all

ERR   Got unexpected memory map file size 37330480, expected 37330472    zm_monitor.cpp	1036

This must be a bug, because when I set LOG_LEVEL_DATABASE=None, stop ZM, truncate zm.Logs table and restart ZM, I'm still getting them. The "Log" tab is gone, but when I manually navigate to /zm/index.php?view=log, I see them coming 2 times per/second.

UPD:
Changed LOG_LEVEL_DATABASE from None to Fatal, waited for some time - no "memory map" messages.
Changed again to Warning - no "memory map" messages and I do get some warnings from zms that should be there.

[iconnor]

Did you restart zm? zmc processes don't pick up logging changes automatically.

[MarsRover]

Did you restart zm? zmc processes don't pick up logging changes automatically.

when I do systemctl restart it restarts the whole CG (cgroup), so yes. I didn't restart the apache part though

UPD:
Just upgraded to yet another new build (1.37.63~20240716103948-focal) on another machine, new strange errors out of the blue, never seen them before:

Code: Select all

zms_exxxx ERR  Invalid frame id specified. 1 > 1     zm_eventstream.cpp:98
web_js ERR Point.fromHex: invalid y coordinate
web_js ERR Invalid seeds, address must fall off the curve
web_js ERR unable to determine span for unrecognized variant