Page 1 of 2
1.34.3 zmc broken with nvidia cuda
Posted: Mon Feb 24, 2020 3:15 pm
by russell_i_brown
I just upgraded from 1.34.2 to 1.34.3 (Ubuntu 18.04.4 LTS, apt-get install zoneminder using
http://ppa.launchpad.net/iconnor/zoneminder-1.34/ubuntu) and zmc broke
I'm using a GPU with self compiled ffmpeg and DecoderHWAccelName set to 'cuda' and DecoderHWAccelDevice set to 'cuvid'.
My zmc logs are full of:
02/24/20 14:10:31.269482 zmc_m1[23525].ERR-zm_ffmpeg_camera.cpp/57 [Failed to get HW surface format for cuda.]
02/24/20 14:10:31.270446 zmc_m1[23525].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was vaapi_vld.]
02/24/20 14:10:31.271070 zmc_m1[23525].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was yuvj420p.]
02/24/20 14:10:31.271646 zmc_m1[23525].ERR-zm_ffmpeg.cpp/548 [Unable to send packet Invalid data found when processing input, continuing]
02/24/20 14:10:31.272176 zmc_m1[23525].WAR-zm_ffmpeg_camera.cpp/982 [Unable to receive frame 0: code -1094995529 Invalid data found when processing input. error count is 0]
and, unsurprisingly, the cameras are not capturing. I changed nothing else apart from the apt update.
As a temporary workaround I've put the 1.32.2 version of zmc in /usr/bin/ and everything's back to working with the GPU.
If it's any help, version 1.34.2 zmc loads (ldd zmc) against:
Code: Select all
libavcodec.so.57 => /usr/lib/x86_64-linux-gnu/libavcodec.so.57
libavcodec.so.58 => /usr/lib/x86_64-linux-gnu/libavcodec.so.58
where the 1.34.3 version loads with the following (no libavcodec.so.58) :
Code: Select all
libavcodec.so.57 => /usr/lib/x86_64-linux-gnu/libavcodec.so.57
Looking at zm_ffmpeg_camera.cpp there does seem to be some checking against the version of libavcodec - is this causing the issue?
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Feb 24, 2020 4:58 pm
by asker
So the 'cuda' and 'cuvid' parts where what worked on my system, Isaac uses a different set of parameters I think. I should add that to the docs that these are not the only values. Isaac uses only `cuda` in the name and has device as empty.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Feb 24, 2020 5:39 pm
by russell_i_brown
Sadly that's not it.
I removed cuvid, reinstated zmc-1.34.3 and the problem still exists
However, running zmc-1.34.2 without cuvid in DecoderHWAccelDevice also seems to work. I wish I understood all this magic!
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Feb 24, 2020 5:48 pm
by asker
Ah I should read posts completely. I missed the part where you said it was an issue after an update. Definitely an Isaac question. Let's wait for him.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Feb 24, 2020 8:15 pm
by iconnor
I will investigate for 1.34.5.
Please note that there really havn't been any changes in that code, so I'm really suspicious.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Feb 24, 2020 9:32 pm
by iconnor
That's pretty weird. I don't see why it would link against two versions of the library.
I just updated my system and it is working fine.
The long and short is that for some reason at run time, it isn't linking against your cuda-enabled libs.
I'm tempted to think that zm is built against the .57 version libs, and the .58 are your cuda-enabled version.
How did you build/get these zm versions? ppa or zmrepo or something else?(Edit: ok you answered that in the first post)
Please note you can get cuda-enabled ffmpeg using
deb
https://zmrepo.zoneminder.com/debian/ffmpeg eoan/
in /etc/apt/sources.list
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Tue Feb 25, 2020 10:58 am
by russell_i_brown
Yes. The .58 is my self compiled ffmpeg 4.1.4.
I tried adding your repo and doing an "apt-get install -t eoan ffmpeg" but it barfs on a whole host of unmet dependencies on my bionic system.
I realise that the GPU stuff is all {b}leading-edge for ZM and getting it to work is like the olden days of package compilation (Nvidia, opencv etc really isn't simple to install) so I'll leave it parked as working for me and revisit when it all becomes a bit more formal.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Mar 02, 2020 9:54 pm
by Ilia_SB
Ah, so I'm not the only one who has that. I'll post here and in the issue I opened on github.
I spent a couple of days recompiling and installing ffmpeg and checking the logs. Here's what I can add:
With ffmpeg 3.4 compiled from sources and installed via checkintall:
Code: Select all
user@nvr:~/FFmpeg$ ldd /usr/bin/zmc | grep libav
libavformat.so.57 => /usr/local/lib/libavformat.so.57 (0x00007f1793e28000)
libavcodec.so.57 => /usr/local/lib/libavcodec.so.57 (0x00007f1792773000)
libavutil.so.55 => /usr/local/lib/libavutil.so.55 (0x00007f17924ed000)
user@nvr:~/FFmpeg$ sudo find / -name libavformat.so*
/usr/local/lib/libavformat.so.57
/usr/local/lib/libavformat.so.57.83.100
/usr/local/lib/libavformat.so
/home/user/FFmpeg/libavformat/libavformat.so.57
/home/user/FFmpeg/libavformat/libavformat.so
ffmpeg version n3.4.7 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
configuration: --extra-cflags=-I/home/user/ffmpeg_build/include --extra-ldflags=-L/home/user/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libx264 --enable-libx265 --enable-shared --enable-gpl --enable-libx265 --enable-nonfree --enable-cuda-sdk --enable-libnpp --enable-nvenc --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
Hardware acceleration methods:
vdpau
vaapi
cuvid
From my understanding I don't have duplicate versions of ffmpeg libs. With DecoderHWAccelName set to cuda ZM reports:
Code: Select all
03/03/20 00:36:09.785002 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/481 [vdpau]
03/03/20 00:36:09.785031 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/481 [cuda]
03/03/20 00:36:09.785058 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/481 [vaapi]
03/03/20 00:36:09.785085 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/488 [Found hwdevice cuda]
03/03/20 00:36:09.785113 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/518 [Selected hw_pix_fmt 129 cuda]
03/03/20 00:36:10.070900 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/526 [Created hwdevice for ]
...
03/03/20 00:36:10.071189 zmc_m1[21628].DB1-zm_ffmpeg.cpp/330 [Dumping codec_context codec_type(0) codec_id(28 h264) width(1920) height(1080) timebase(1/180000) format(yuvj420p) gop_size 12 max_b_frames 0 me_cmp 0 me_range 0 qmin 2 qmax 31]
03/03/20 00:36:10.071202 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/559 [HWACCEL in use]
...
03/03/20 00:36:10.072293 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/57 [Failed to get HW surface format for cuda.]
03/03/20 00:36:10.078082 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was vdpau.]
03/03/20 00:36:10.085462 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was vaapi_vld.]
03/03/20 00:36:10.095021 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was yuvj420p.]
Now, apt remove ffmpeg, build latest ffmpeg, checkinstall, ldconfig:
Code: Select all
user@nvr:~/FFmpeg$ ldd /usr/bin/zmc | grep libav
libavformat.so.57 => not found
libavcodec.so.57 => not found
libavutil.so.55 => not found
libavutil.so.55 => not found
libavutil.so.55 => not found
user@nvr:~/FFmpeg$ sudo find / -name libavformat.so*
/usr/local/lib/libavformat.so.58.39.101
/usr/local/lib/libavformat.so
/usr/local/lib/libavformat.so.58
/home/user/FFmpeg/libavformat/libavformat.so
/home/user/FFmpeg/libavformat/libavformat.so.58
user@nvr:~/FFmpeg$ sudo zmc -m1
zmc: error while loading shared libraries: libavformat.so.57: cannot open shared object file: No such file or directory
user@nvr:~/FFmpeg$ ffmpeg -hwaccels
ffmpeg version N-96863-g9b22254331 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
configuration: --extra-cflags=-I/home/user/ffmpeg_build/include --extra-ldflags=-L/home/user/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libx264 --enable-libx265 --enable-shared --enable-gpl --enable-libx265 --enable-nonfree --enable-cuda-sdk --enable-libnpp --enable-nvenc --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
libavutil 56. 42.100 / 56. 42.100
libavcodec 58. 73.102 / 58. 73.102
libavformat 58. 39.101 / 58. 39.101
libavdevice 58. 9.103 / 58. 9.103
libavfilter 7. 77.100 / 7. 77.100
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
libpostproc 55. 6.100 / 55. 6.100
Hardware acceleration methods:
vdpau
cuda
vaapi
cuvid
Hope that helps.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Thu Apr 16, 2020 9:11 am
by zemtos
Hi llia_SB,
I have exact same problem.
I built the ffmpeg and it supports cuda, cuvid already, but I do not know how to tell zmc to link to that libraries instead of the libav*.so.57 and libavcodec.so.55. May you tell me how to do these steps in details.
Now, apt remove ffmpeg, build latest ffmpeg, checkinstall, ldconfig:
Thanks.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon Apr 20, 2020 4:29 am
by zemtos
Finally I got it working, but on Ubuntu 19.10. Install Ubuntu 19.10, rebuild ffmpeg with cuda support:
Code: Select all
./configure --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --prefix=/usr --disable-static --enable-shared --enable-cuda-nvcc --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --enable-cuda --enable-libass --enable-libfdk-aac --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-gpl --enable-nonfree --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
sudo make -j4
sudo make install
It will overwrite the default installation libav*.so* of Ubuntu 19.10.
It will take approx 93MB of GPU RAM for 1280x720p @ 12fps by nvidia-smi report.
0 14216 C /usr/bin/zmc 93MiB
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Tue May 12, 2020 5:31 pm
by Helipil0t
So I've successfully re-compiled ffmpeg with cuda support on Ubuntu 20.04 and am still getting errors in my logs. Using "watch nvidia-smi" I'm able to see zmc process come on and off. Meanwhile my logs are spitting out all kinds of errors.
Ubuntu Focal 20.04
ZoneMinder Version: 1.34.12
Nvidia 440.64 / Cuda 10.2
ESXi with GPU PCI passthrough
Logs:
Code: Select all
2020-05-12 13:24:10.783538 zmc_m1[5075].WAR-zm_ffmpeg_camera.cpp/969 [Unable to receive frame 0: code -1094995529 Invalid data found when processing input. error count is 0]
2020-05-12 13:24:10.782937 zmc_m1[5075].ERR-zm_ffmpeg.cpp/547 [Unable to send packet Invalid data found when processing input, continuing]
2020-05-12 13:24:10.782487 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was yuvj420p.]
2020-05-12 13:24:10.782042 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was yuvj420p.]
2020-05-12 13:24:10.781599 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was vaapi_vld.]
2020-05-12 13:24:10.781006 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was vdpau.]
2020-05-12 13:24:10.780188 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/56 [Failed to get HW surface format for cuda.]
I tried a hashcat benchmark just to make sure my cuda drivers were install properly and everything seems to be working fine.
Anyone have any ideas?
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Wed May 13, 2020 5:49 pm
by Helipil0t
Just a follow up to my problem. I wanted to make sure ffmpeg was working properly so I tried the following:
Code: Select all
ffmpeg -hwaccel cuda -i input -f null -
And it runs but with erros: nvidia-smi shows ffmpeg using about 81 MB of video ram.
It decodes but then shows something along the lines of "error loading libnvcuvid.so.1"
So then I tried installing nvidia-driver-440 and installed the missing libraries.
Code: Select all
sudo apt install nvidia-driver-440
I was running the headless version of the nvidia drivers and I'm guessing they don't include the same libraries.
Anyway long story short everything is running perfectly now.
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Mon May 18, 2020 10:46 am
by JimNoble
I'm also seeing this.
I have a repo-installed version 4.2.2 of ffmpeg, which provides libavformat.so.58, and zmc (v1.34.14) is insisting on linking to libavformat.so.57.
I've been wondering if the problem could be that zoneminder is looking for libavformat version 58 with a minor version >= 64, where as all the release versions of ffmpeg have a libavformat with a minor version of 29?
https://github.com/ZoneMinder/zoneminde ... eg.cpp#L87
(Also documented here:)
https://github.com/ZoneMinder/zoneminde ... -630098352
(I'm a H/W guy though, so maybe "It doesn't work like that"...
)
James
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Fri Jul 03, 2020 10:34 pm
by jds_1981
I've just been trying to enable cuda but I encounter the issues mentioned above. I suppose there's no good solution yet?
Re: 1.34.3 zmc broken with nvidia cuda
Posted: Sun Jul 05, 2020 8:55 am
by jds_1981
russell_i_brown wrote: ↑Mon Feb 24, 2020 3:15 pm
I just upgraded from 1.34.2 to 1.34.3 (Ubuntu 18.04.4 LTS, apt-get install zoneminder using
http://ppa.launchpad.net/iconnor/zoneminder-1.34/ubuntu) and zmc broke
I'm using a GPU with self compiled ffmpeg and DecoderHWAccelName set to 'cuda' and DecoderHWAccelDevice set to 'cuvid'.
~~~~snip~~~~
As a temporary workaround I've put the 1.32.2 version of zmc in /usr/bin/ and everything's back to working with the GPU.
How do I get the 1.32.2 binary version? Or did you specify this version for install?