Direct Writing, OOM Crashes

[Quantum]

After having failed to make ZM work in Podman containers, I've now built it in CentOS Stream 9. It seems to be working, although it's running max CPU with 4 cores and 4 cameras.

I want to set it so that zmc does direct writing of the transport stream to disk but am unclear on how. In a camera's Settings|Storage there's Save Jpegs=Frame+Analysis, which seems to be the default, and there's Video Writer=Disabled which is the setting I suspect that I need to set to Camera Passthrough. Is this correct? Is Jpeg=Frame+Analysis advisable?

Three of my cameras support H265 now. Does this work with ZM yet?

Also the zoneminder service keeps getting killed after 60 seconds or so, on OOM. The virtual machine has 8192GB allocated. What does it want?

My plan is to install and get AI Event Notification running instead of zma.


Edit: Well I've set Video Writer=Camera Passthrough, and now the zoneminder.service gives an endless string of:
Dec 19 07:11:36 reticulum.darkmatter.org zmc_m3[4238]: ERR [zmc_m3] [Error writing packet: Invalid argument]

Rights on the destination parts are set correctly and selinux is Permissive.

Additionally zmc is consuming tons of CPU, 84.1%-41.1%:
4242 nginx 20 0 5228576 3.5g 416384 S 84.1 23.0 2:47.02 zmc
4238 nginx 20 0 5347892 3.5g 827520 S 72.5 23.0 2:30.95 zmc
2268 root 20 0 1382640 61744 38044 R 68.2 0.4 5:33.60 Xorg
4230 nginx 20 0 6019772 4.8g 416512 R 42.4 31.5 1:31.23 zmc
4234 nginx 20 0 2185604 1.2g 324608 S 41.1 7.6 1:25.38 zmc
... and zma is nowhere to be seen.

I don't understand what's going on here.

[d4v1d]

If you have an H264 source stream, you need to set video writer to passtrough, for MJPEG streams you need to set it to encode.

I have also experienced high CPU load with increasing RAM usage over time.

Disabling the save jpegs option decreased my CPU load by half.

It also seems that a high log level to the database somehow causes the image buffer queues to grow. I don't know how this is connected to the database writes but disabling it or setting it to at least -1 improves the memory behaviour of ZoneMinder.

There is a memory leak in the audio record path in version 1.36, so increasing RAM usage can also be connected to this issue. So audio record has to be disabled.

[Quantum]

If you have an H264 source stream, you need to set video writer to passtrough

Is this done the way I propose above?

I have also experienced high CPU load with increasing RAM usage over time.

This sounds like a memory leak. If it keeps happening I guess I'll have to resort to compiling ZM.

Disabling the save jpegs option decreased my CPU load by half.

I've now done this. I don't know the point of saving the jpegs anyway.

It also seems that a high log level to the database somehow causes the image buffer queues to grow. I don't know how this is connected to the database writes but disabling it or setting it to at least -1 improves the memory behaviour of ZoneMinder.

Mine was set to Info, but I've now disabled database logging. This is prob not a good idea.

There is a memory leak in the audio record path in version 1.36, so increasing RAM usage can also be connected to this issue. So audio record has to be disabled.

Certainly don't want to turn off audio recording.

[d4v1d]

Is this done the way I propose above?

Actually I've never tested H265 streams myself, but with your errors it seems that passthrough is not correct option.

This sounds like a memory leak. If it keeps happening I guess I'll have to resort to compiling ZM.

I don't think it's a memory leak because it does still keep the pointers to the allocated memory. As I already mentioned I think the increasing memory is connected to the packet queues but I still don't really understand under what circumstances they fill up. Another thing I experienced is when I compile ZoneMinder with optimizations on (-O2) the queues tend to fill up much faster than in debug mode (-O0). Maybe @iconnor knows how this could be connected?

Mine was set to Info, but I've now disabled database logging. This is prob not a good idea.

You can still have a higher log level to file or to syslog.

Certainly don't want to turn off audio recording.

In this case, you have compile ZoneMinder yourself and apply the following fix:
https://github.com/ZoneMinder/zoneminde ... 3787/files
or just use the current version on master branch where this fix is already applied.

[Quantum]

Is this done the way I propose above?

Actually I've never tested H265 streams myself, but with your errors it seems that passthrough is not correct option.

This sounds like a memory leak. If it keeps happening I guess I'll have to resort to compiling ZM.

I don't think it's a memory leak because it does still keep the pointers to the allocated memory. As I already mentioned I think the increasing memory is connected to the packet queues but I still don't really understand under what circumstances they fill up. Another thing I experienced is when I compile ZoneMinder with optimizations on (-O2) the queues tend to fill up much faster than in debug mode (-O0). Maybe @iconnor knows how this could be connected?

It appears that you're right. With the above changes I've just made it through the night without getting killed by OOM daemon.

I'm still not clear on how to enable direct passthrough recording to disk. I'll enable H265 on one of my cameras. If I remember correctly, years ago we could not do this because H265 was not supported by ffmpeg. Maybe it is now. I'll install x265, libde265, and x265-libs.

Mine was set to Info, but I've now disabled database logging. This is prob not a good idea.

You can still have a higher log level to file or to syslog.

I'll re-enable at Info level.

[Quantum]

At this point it is clear that ZM can capture, direct write, and analyze H-265, as long as the correct packages are installed for ffmpeg. (above)

However zoneminder.service while running constantly gets "ERR [zmc_m3] [Error writing packet: Invalid argument]".

zmc_m1.log and zmc_m4.log are mostly capturing, but have a few of these:

Code: Select all

12/21/23 07:19:10.402697 zmc_m1[113932].WAR-zm_videostore.cpp/1398 [non increasing dts, fixing. our dts 13001670 stream 0 last_dts 13005180. reorder_queue_size=0]
12/21/23 07:19:10.402812 zmc_m1[113932].WAR-zm_videostore.cpp/1409 [pkt.dts(13005180) must be <= pkt.pts(13001670).Decompression must happen before presentation.]
12/21/23 07:19:10.402834 zmc_m1[113932].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/21/23 07:19:10.410529 zmc_m1[113932].WAR-zm_videostore.cpp/1398 [non increasing dts, fixing. our dts 13004730 stream 0 last_dts 13005180. reorder_queue_size=0]
12/21/23 07:19:10.410624 zmc_m1[113932].WAR-zm_videostore.cpp/1409 [pkt.dts(13005180) must be <= pkt.pts(13004730).Decompression must happen before presentation.]
12/21/23 07:19:10.410652 zmc_m1[113932].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]

zmc_m2 has no errors (H-264).

ZMC_m3 is just full of these, only 1 or 2 lines of capturing, whether on H-264H or H-265, and with SubStream off:

Code: Select all

12/21/23 07:25:23.346066 zmc_m3[113938].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/21/23 07:25:23.436018 zmc_m3[113938].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/21/23 07:25:23.436070 zmc_m3[113938].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]

This is the only Lorex (LNE8974AB) I have and there is no firmware update. I've changed every setting in the camera and no difference. Oddly the wiki Troubleshooting does not say how to put ZM into debug mode, and G**gle is baffled, so this is all I know.

[dougmccrary]

Have you seen this?
https://zoneminder.readthedocs.io/en/1. ... gging.html

[d4v1d]

Hi Quantum,

what if you directly try to capture the input streams with FFMPEG:

Code: Select all

ffmpeg -i <input_url> /tmp/test.mp4

[Quantum]

Have you seen this?
https://zoneminder.readthedocs.io/en/1. ... gging.html

Thx. Wow, I never would have found that.

The only sign of trouble in the m3 log is:

Code: Select all

12/22/23 07:50:02.326420 zmc_m3[143303].DB1-zm_db.cpp/190 [Success running sql query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1703260202.322806, 'zmc_m3', 0, 143353, -1, 'WAR', 'Application provided invalid, non monotonically increasing dts to muxer in stream 1: 1920 >= 1920', 'zm_ffmpeg.cpp', 69 )]
12/22/23 07:50:02.328743 zmc_m3[143303].DB1-zm_db.cpp/190 [Success running sql query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1703260202.323485, 'zmc_m3', 0, 143353, -2, 'ERR', 'Error writing packet: Invalid argument', 'zm_videostore.cpp', 1425 )]
12/22/23 07:50:02.329370 zmc_m3[143352].DB1-zm_zone_stats.h/54 [ZoneStat: GetStats zone_id: 3 pixel_diff=32 alarm_pixels=2 alarm_filter_pixels=0 alarm_blob_pixels=0 alarm_blobs=0 min_blob_size=0 max_blob_size=0 alarm_box=(0,0=>0,0) alarm_center=(0,0) score=0]
12/22/23 07:50:02.329389 zmc_m3[143352].DB1-zm_zone_stats.h/54 [ZoneStat: After detect motion zone_id: 3 pixel_diff=32 alarm_pixels=2 alarm_filter_pixels=0 alarm_blob_pixels=0 alarm_blobs=0 min_blob_size=0 max_blob_size=0 alarm_box=(0,0=>0,0) alarm_center=(0,0) score=0]

I don't understand it.

Hi Quantum,

what if you directly try to capture the input streams with FFMPEG:

Code: Select all

ffmpeg -i <input_url> /tmp/test.mp4

I don't get it. I 'touch'ed the file and made it world-writable. selinux is permissive.

Code: Select all

reticulum:/etc# ffmpeg -i rtsp://UN:PW@10.2.5.52/cam/realmonitor?channel=1&subtype=01 /tmp/test.mp4
[1] 141506
****-bash: /tmp/test.mp4: Permission denied****
reticulum:/etc# ffmpeg version 5.1.4 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio --enable-libdrm --enable-libjack --enable-libjxl --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libshaderc --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzvbi --enable-lv2 --enable-avfilter --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100

It pretends to run, but the file stays at zero bytes. "Permission denied"

I thought it might be because monitor3 is H.265 so I made a test.mkv file, but same drill: "Permission denied".

[mikb]

Code: Select all

ffmpeg -i <input_url> /tmp/test.mp4

Code: Select all

reticulum:/etc# ffmpeg -i rtsp://UN:PW@10.2.5.52/cam/realmonitor?channel=1&subtype=01 /tmp/test.mp4
[1] 141506
****-bash: /tmp/test.mp4: Permission denied****

I don't know SELinux. I do know "permission denied" means that ain't gonna work

For some reason, your OS is saying you cannot create/update a file in /tmp -- maybe the security is set more paranoid than my system (which usually allows files to be created in /tmp but not EXECUTED there ...)

Instead of /tmp/whatever.mp4, try letting FFMPEG create a file somewhere you do have proper privileges (a /home.your-user directory for example)

Any amount of fiddling with the file format and FFMPEG won't get you out of that one, I think

[Quantum]

Ya selinux is disabled when set to Permissive.
I'm doing all of this as root.

I find that in VLC I can Open Network Stream and watch the offending camera RTSP stream just fine. Same with my other monitors.

I suspicion that the 'improper setting' is some ZM setting incompatible with the camera. The only clue I have is in the debug log above. Idk what to do.


Edit: Trying to restart ZM I'm getting firewall blocks on UDP 45556 and 35074 on mon 3. No one knows what these are for.

[dougmccrary]

This cam is UDP? Then "Maximum Image Buffer Size (frames)" should be 0 (unlimited), or 2xKey frame interval + 1.

[Quantum]

Nah, all 4 cams are TCP.

I am out of ideas. On m3 just billions of:

Code: Select all

12/23/23 00:00:12.919244 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:12.919407 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:12.944484 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:12.944614 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:12.978309 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.058028 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.058115 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.091631 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.161466 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.161599 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.192368 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.266974 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.283341 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.288073 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.315918 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.315994 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.316010 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.444702 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.469046 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.505517 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.530169 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.584872 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.584956 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.613173 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]
12/23/23 00:00:13.643264 zmc_m3[164730].ERR-zm_videostore.cpp/1425 [Error writing packet: Invalid argument]

[mikb]

I'm doing all of this as root.

Wow. Then you need to give your computer a stern talking to.

"I am root. What do you 'Permission Denied!' Who's in charge here?"

Still, your test ffmpeg command should be happier if you give it a destination that it can write to. If the computer will allow you to, obviously. Maybe in /root/test.mp4 if you are root. Dunno how much space you have there, watch out for filling it up

If you were doing this in some sort of container/sandbox/VM then I could understand that maybe something super-paranoid/secure was making parts of the filesystem read-only from outside of your control. Step out of the VM/etc. and give it a gentle smack, then try again ...

Even if there was some horrible underlying fault, e.g. a corrupted file system, you usually get a better clue from e.g. can't write to /tmp because -- "file system is read only", resulting from the kernel realising "the file system is broken, let's stop ALL writing, even root, to protect things". Or "No space left on device" because something went wrong and filled /tmp to bursting.

[Quantum]

Ok this works just fine. Saves to the file (in H.265) on the monitor which is making errors in ZM, and the file plays back fine in VLC:

Code: Select all

# ffmpeg -hide_banner -y -loglevel error -rtsp_transport tcp -use_wallclock_as_timestamps 1 -i "rtsp://UN:PW@10.2.5.52/cam/realmonitor?channel=1&subtype=0" -vcodec copy -acodec copy -f segment -reset_timestamps 1 -segment_time 900 -segment_format mkv -segment_atclocktime 1 -strftime 1 %Y%m%dT%H%M%S.mkv

This confirms to me that there is some setting in ZM which has a problem.