1.34.3 zmc broken with nvidia cuda

[russell_i_brown]

I just upgraded from 1.34.2 to 1.34.3 (Ubuntu 18.04.4 LTS, apt-get install zoneminder using http://ppa.launchpad.net/iconnor/zoneminder-1.34/ubuntu) and zmc broke

I'm using a GPU with self compiled ffmpeg and DecoderHWAccelName set to 'cuda' and DecoderHWAccelDevice set to 'cuvid'.

My zmc logs are full of:

02/24/20 14:10:31.269482 zmc_m1[23525].ERR-zm_ffmpeg_camera.cpp/57 [Failed to get HW surface format for cuda.]
02/24/20 14:10:31.270446 zmc_m1[23525].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was vaapi_vld.]
02/24/20 14:10:31.271070 zmc_m1[23525].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was yuvj420p.]
02/24/20 14:10:31.271646 zmc_m1[23525].ERR-zm_ffmpeg.cpp/548 [Unable to send packet Invalid data found when processing input, continuing]
02/24/20 14:10:31.272176 zmc_m1[23525].WAR-zm_ffmpeg_camera.cpp/982 [Unable to receive frame 0: code -1094995529 Invalid data found when processing input. error count is 0]

and, unsurprisingly, the cameras are not capturing. I changed nothing else apart from the apt update.

As a temporary workaround I've put the 1.32.2 version of zmc in /usr/bin/ and everything's back to working with the GPU.

If it's any help, version 1.34.2 zmc loads (ldd zmc) against:

Code: Select all

 
libavcodec.so.57 => /usr/lib/x86_64-linux-gnu/libavcodec.so.57 
libavcodec.so.58 => /usr/lib/x86_64-linux-gnu/libavcodec.so.58 

where the 1.34.3 version loads with the following (no libavcodec.so.58) :

Code: Select all

libavcodec.so.57 => /usr/lib/x86_64-linux-gnu/libavcodec.so.57 

Looking at zm_ffmpeg_camera.cpp there does seem to be some checking against the version of libavcodec - is this causing the issue?

[asker]

So the 'cuda' and 'cuvid' parts where what worked on my system, Isaac uses a different set of parameters I think. I should add that to the docs that these are not the only values. Isaac uses only `cuda` in the name and has device as empty.

[russell_i_brown]

Sadly that's not it.

I removed cuvid, reinstated zmc-1.34.3 and the problem still exists

However, running zmc-1.34.2 without cuvid in DecoderHWAccelDevice also seems to work. I wish I understood all this magic!

[asker]

Ah I should read posts completely. I missed the part where you said it was an issue after an update. Definitely an Isaac question. Let's wait for him.

[iconnor]

I will investigate for 1.34.5.

Please note that there really havn't been any changes in that code, so I'm really suspicious.

[iconnor]

That's pretty weird. I don't see why it would link against two versions of the library.

I just updated my system and it is working fine.

The long and short is that for some reason at run time, it isn't linking against your cuda-enabled libs.

I'm tempted to think that zm is built against the .57 version libs, and the .58 are your cuda-enabled version.

How did you build/get these zm versions? ppa or zmrepo or something else?(Edit: ok you answered that in the first post)

Please note you can get cuda-enabled ffmpeg using
deb https://zmrepo.zoneminder.com/debian/ffmpeg eoan/

in /etc/apt/sources.list

[russell_i_brown]

Yes. The .58 is my self compiled ffmpeg 4.1.4.

I tried adding your repo and doing an "apt-get install -t eoan ffmpeg" but it barfs on a whole host of unmet dependencies on my bionic system.

I realise that the GPU stuff is all {b}leading-edge for ZM and getting it to work is like the olden days of package compilation (Nvidia, opencv etc really isn't simple to install) so I'll leave it parked as working for me and revisit when it all becomes a bit more formal.

[Ilia_SB]

Ah, so I'm not the only one who has that. I'll post here and in the issue I opened on github.
I spent a couple of days recompiling and installing ffmpeg and checking the logs. Here's what I can add:
With ffmpeg 3.4 compiled from sources and installed via checkintall:

Code: Select all

user@nvr:~/FFmpeg$ ldd /usr/bin/zmc | grep libav
        libavformat.so.57 => /usr/local/lib/libavformat.so.57 (0x00007f1793e28000)
        libavcodec.so.57 => /usr/local/lib/libavcodec.so.57 (0x00007f1792773000)
        libavutil.so.55 => /usr/local/lib/libavutil.so.55 (0x00007f17924ed000)

user@nvr:~/FFmpeg$ sudo find / -name libavformat.so*
/usr/local/lib/libavformat.so.57
/usr/local/lib/libavformat.so.57.83.100
/usr/local/lib/libavformat.so
/home/user/FFmpeg/libavformat/libavformat.so.57
/home/user/FFmpeg/libavformat/libavformat.so

ffmpeg version n3.4.7 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --extra-cflags=-I/home/user/ffmpeg_build/include --extra-ldflags=-L/home/user/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libx264 --enable-libx265 --enable-shared --enable-gpl --enable-libx265 --enable-nonfree --enable-cuda-sdk --enable-libnpp --enable-nvenc --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Hardware acceleration methods:
vdpau
vaapi
cuvid

From my understanding I don't have duplicate versions of ffmpeg libs. With DecoderHWAccelName set to cuda ZM reports:

Code: Select all

03/03/20 00:36:09.785002 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/481 [vdpau]
03/03/20 00:36:09.785031 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/481 [cuda]
03/03/20 00:36:09.785058 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/481 [vaapi]
03/03/20 00:36:09.785085 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/488 [Found hwdevice cuda]
03/03/20 00:36:09.785113 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/518 [Selected hw_pix_fmt 129 cuda]
03/03/20 00:36:10.070900 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/526 [Created hwdevice for ]
...
03/03/20 00:36:10.071189 zmc_m1[21628].DB1-zm_ffmpeg.cpp/330 [Dumping codec_context codec_type(0) codec_id(28 h264) width(1920) height(1080)  timebase(1/180000) format(yuvj420p) gop_size 12 max_b_frames 0 me_cmp 0 me_range 0 qmin 2 qmax 31]
03/03/20 00:36:10.071202 zmc_m1[21628].DB1-zm_ffmpeg_camera.cpp/559 [HWACCEL in use]
...
03/03/20 00:36:10.072293 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/57 [Failed to get HW surface format for cuda.]
03/03/20 00:36:10.078082 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was vdpau.]
03/03/20 00:36:10.085462 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was vaapi_vld.]
03/03/20 00:36:10.095021 zmc_m1[21628].ERR-zm_ffmpeg_camera.cpp/60 [Available HW surface format was yuvj420p.]

Now, apt remove ffmpeg, build latest ffmpeg, checkinstall, ldconfig:

Code: Select all

user@nvr:~/FFmpeg$ ldd /usr/bin/zmc | grep libav
        libavformat.so.57 => not found
        libavcodec.so.57 => not found
        libavutil.so.55 => not found
        libavutil.so.55 => not found
        libavutil.so.55 => not found
        
user@nvr:~/FFmpeg$ sudo find / -name libavformat.so*
/usr/local/lib/libavformat.so.58.39.101
/usr/local/lib/libavformat.so
/usr/local/lib/libavformat.so.58
/home/user/FFmpeg/libavformat/libavformat.so
/home/user/FFmpeg/libavformat/libavformat.so.58

user@nvr:~/FFmpeg$ sudo zmc -m1
zmc: error while loading shared libraries: libavformat.so.57: cannot open shared object file: No such file or directory

user@nvr:~/FFmpeg$ ffmpeg -hwaccels
ffmpeg version N-96863-g9b22254331 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
  configuration: --extra-cflags=-I/home/user/ffmpeg_build/include --extra-ldflags=-L/home/user/ffmpeg_build/lib --extra-libs='-lpthread -lm' --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libx264 --enable-libx265 --enable-shared --enable-gpl --enable-libx265 --enable-nonfree --enable-cuda-sdk --enable-libnpp --enable-nvenc --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
  libavutil      56. 42.100 / 56. 42.100
  libavcodec     58. 73.102 / 58. 73.102
  libavformat    58. 39.101 / 58. 39.101
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 77.100 /  7. 77.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Hardware acceleration methods:
vdpau
cuda
vaapi
cuvid

Hope that helps.

[zemtos]

Hi llia_SB,

I have exact same problem.
I built the ffmpeg and it supports cuda, cuvid already, but I do not know how to tell zmc to link to that libraries instead of the libav*.so.57 and libavcodec.so.55. May you tell me how to do these steps in details.

Now, apt remove ffmpeg, build latest ffmpeg, checkinstall, ldconfig:

Thanks.

[zemtos]

Finally I got it working, but on Ubuntu 19.10. Install Ubuntu 19.10, rebuild ffmpeg with cuda support:

Code: Select all

./configure --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --prefix=/usr --disable-static --enable-shared --enable-cuda-nvcc --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --enable-cuda --enable-libass --enable-libfdk-aac --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-gpl --enable-nonfree --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64

sudo make -j4
sudo make install

It will overwrite the default installation libav*.so* of Ubuntu 19.10.

It will take approx 93MB of GPU RAM for 1280x720p @ 12fps by nvidia-smi report.

0 14216 C /usr/bin/zmc 93MiB

[Helipil0t]

So I've successfully re-compiled ffmpeg with cuda support on Ubuntu 20.04 and am still getting errors in my logs. Using "watch nvidia-smi" I'm able to see zmc process come on and off. Meanwhile my logs are spitting out all kinds of errors.

Ubuntu Focal 20.04
ZoneMinder Version: 1.34.12
Nvidia 440.64 / Cuda 10.2
ESXi with GPU PCI passthrough

Logs:

Code: Select all

2020-05-12 13:24:10.783538 zmc_m1[5075].WAR-zm_ffmpeg_camera.cpp/969 [Unable to receive frame 0: code -1094995529 Invalid data found when processing input. error count is 0]
2020-05-12 13:24:10.782937 zmc_m1[5075].ERR-zm_ffmpeg.cpp/547 [Unable to send packet Invalid data found when processing input, continuing]
2020-05-12 13:24:10.782487 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was yuvj420p.]
2020-05-12 13:24:10.782042 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was yuvj420p.]
2020-05-12 13:24:10.781599 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was vaapi_vld.]
2020-05-12 13:24:10.781006 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/59 [Available HW surface format was vdpau.]
2020-05-12 13:24:10.780188 zmc_m1[5075].ERR-zm_ffmpeg_camera.cpp/56 [Failed to get HW surface format for cuda.]

I tried a hashcat benchmark just to make sure my cuda drivers were install properly and everything seems to be working fine.
Anyone have any ideas?

[Helipil0t]

Just a follow up to my problem. I wanted to make sure ffmpeg was working properly so I tried the following:

Code: Select all

ffmpeg -hwaccel cuda -i input -f null -

And it runs but with erros: nvidia-smi shows ffmpeg using about 81 MB of video ram.
It decodes but then shows something along the lines of "error loading libnvcuvid.so.1"
So then I tried installing nvidia-driver-440 and installed the missing libraries.

Code: Select all

sudo apt install nvidia-driver-440

I was running the headless version of the nvidia drivers and I'm guessing they don't include the same libraries.

Anyway long story short everything is running perfectly now.

[JimNoble]

I'm also seeing this.

I have a repo-installed version 4.2.2 of ffmpeg, which provides libavformat.so.58, and zmc (v1.34.14) is insisting on linking to libavformat.so.57.

I've been wondering if the problem could be that zoneminder is looking for libavformat version 58 with a minor version >= 64, where as all the release versions of ffmpeg have a libavformat with a minor version of 29?

https://github.com/ZoneMinder/zoneminde ... eg.cpp#L87

(Also documented here:)
https://github.com/ZoneMinder/zoneminde ... -630098352

(I'm a H/W guy though, so maybe "It doesn't work like that"... )

James

[jds_1981]

I've just been trying to enable cuda but I encounter the issues mentioned above. I suppose there's no good solution yet?

[jds_1981]

I just upgraded from 1.34.2 to 1.34.3 (Ubuntu 18.04.4 LTS, apt-get install zoneminder using http://ppa.launchpad.net/iconnor/zoneminder-1.34/ubuntu) and zmc broke

I'm using a GPU with self compiled ffmpeg and DecoderHWAccelName set to 'cuda' and DecoderHWAccelDevice set to 'cuvid'.

~~~~snip~~~~


As a temporary workaround I've put the 1.32.2 version of zmc in /usr/bin/ and everything's back to working with the GPU.

How do I get the 1.32.2 binary version? Or did you specify this version for install?