Zoneminder and facedetection hook overall performance and detection issues

Forum for questions and support relating to the 1.32.x releases only.
Post Reply
fulvioth
Posts: 21
Joined: Wed Dec 18, 2019 5:27 pm

Zoneminder and facedetection hook overall performance and detection issues

Post by fulvioth »

Hello all!
this is my first post so i wanna greet all and say that the original author and all the contributors have made an amazing job! I'm true :)
It's about two weeks now i'm working on setting up my first zm server so i'm not an expert but i swear i read all the official guides and wikis and forums many times and ended up with the following results (or better goals for me hehe). I still have a couple of issues i was not able to solve so i really hope someone can help.

But first a general description of the environment:
[Workstation]
Ubunto 18.04
CPU Intel Q9400 2.66Ghz quad cpu
6Gb ram, 70% mapped memory
NVidia GeForce 8400
No other relevant loads beside zoneminder

[Zoneminder]
Zoneminder 1.32.3 official release, using https with letsencrypt cert
EventServer 4.7 with its object detection/face recognition hook (hog actually)
ZmNinja 1.3.082 android
Camera1: Onvif 1920x1080 10fps, wifi
ffmpeg, modect, buffersize 100, Pre/Post image count 50
Camera2: Onvif 1280x960 15fps, lan cable, PTZ (via onvif)
ffmpeg, modect, buffersize 100, Pre/Post image count 50
Camera3: Onvif 1280x960 15fps, wifi, PTZ (via onvif)
ffmpeg, modect, buffersize 100, Pre/Post image count 50

[General stats]
Load (from zm web): 1-3 as an average (this is the highest load i saw but havent checked a whole day)

Code: Select all

df -h
Filesystem…..Size  Used Avail Use% Mounted on
tmpfs…………..4.1G  1.7G  2.4G  42% /dev/shm
Web interface is pretty smooth and streaming, alarms, face detection is ok. I'm running it from a win10 machine with Firefox. LAN and WAN access work

But… there's always a but… more than one tbh.
Logs are set to debug for zmdetect, zmeventnotification
1) Object/face detection stops working after restarting zoneminder service
When i restart the workstation (or just the zoneminder service via systemctl), object detection stops working. after some time (about 1 hour) everything fix ups by itself. Here's the relevant part of the error (ful logs attached)

Code: Select all

CONSOLE INFO:2019-12-17,23:16:37 Invoking hook:'/var/lib/zmeventnotification/bin/zm_detect_wrapper.sh' 1078 1 "Living room" "Motion Alarm,"
Traceback (most recent call last):
  File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/usr/lib/python3.6/http/client.py", line 1407, in connect
    super().connect()
  File "/usr/lib/python3.6/http/client.py", line 946, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/zmeventnotification/bin/zm_detect.py", line 75, in <module>
    filename1, filename2, filename1_bbox, filename2_bbox = utils.download_files(args)
  File "/usr/local/lib/python3.6/dist-packages/zmes_hook_helpers/utils.py", line 129, in download_files
    input_file = opener.open(url)
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
i guess it's when it tries to download the alarm image. i didn't find details about the url in this log. But the addresses to get snapshot and alarmed frame work. I tested it following the official guide and accessing the frame related to this exception (from the win machine).
Apache logs are clean, i only see lots of [Login successful for user "admin"]. Is this normal?
As i said after some time it simply starts working, the error disappears and face detection completes.
i'm not sure if times match but, can the uninstallation of another nvr sw and node.js be the cause? i've paid attention, followed the guides and everything else seems ok.
When this error happens, i don't get notifications sent. No matter if its a true alarm or forced one.
I don't know if this error happens when zm restarts itself every 3hrs. Is it the same as restarting the service from command line?

2) Face detection and cnn model performance
With this configuration cnn model takes something like 150-200s to complete the analysis. This is why im using hog that takes just 2-3s (for face detection).
Do you think that these high cnn times are normal with my hw/sw configuration? They sound weird to me but i don't really know...
Is there a way to keep the hog model in memory thus saving the initial 3-4s initialization?

I would really appreciate any advice for configuration changes in my environment or your overall opinion
i have another issue with zmNinja but i will open another thread for it. Perhaps it could be somehow related to this. Hope someone can help
Thanks in advance!
zmlogs.zip
(316.46 KiB) Downloaded 263 times
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by asker »

Brief responses:

1. No, 100+ seconds is completely out of line. On my system when I did not have a GPU, I got around 5 seconds for a 600px wide image after I enabled cpu optimizations and libblas etc. . With GPU its less than a second (1050i) (see this. Check the resize value in objectconfig.ini and make sure its not bigger than 1600 (even so, 100+ second is out of line)

2. Since you have a GPU, you should be compiling/installing dlib with GPU support. Before that you need to make sure you have CUDA drivers installed. My instructions on how I updated my libraries to support GPU are here

3. detect getting stuck at downloading files after ZM restarts - no idea. Haven't seen that happen before. Your logs show "Connection refused" when detect tries to download an image.
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
fulvioth
Posts: 21
Joined: Wed Dec 18, 2019 5:27 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by fulvioth »

Hi asker, thanks for your tips
1. No, 100+ seconds is completely out of line. On my system when I did not have a GPU, I got around 5 seconds for a 600px wide image after I enabled cpu optimizations and libblas etc. . With GPU its less than a second (1050i) (see this. Check the resize value in objectconfig.ini and make sure its not bigger than 1600 (even so, 100+ second is out of line)
resize value is 1200, i followed a guide and i did install nvidia driver, libblas, then dlib and checked that it found libblas during install and then the face recognition… but… ther's always a but hehe
2. Since you have a GPU, you should be compiling/installing dlib with GPU support. Before that you need to make sure you have CUDA drivers installed. My instructions on how I updated my libraries to support GPU are here
i really have not memory the guide did mention cuda drivers. so that could be the missing point. i checked my system and i found:

Code: Select all

root@home:/etc/zm# locate cuda
/usr/bin/nvidia-cuda-mps-control
/usr/bin/nvidia-cuda-mps-server
/usr/include/linux/cuda.h
/usr/lib/i386-linux-gnu/libcuda.so
/usr/lib/i386-linux-gnu/libcuda.so.1
/usr/lib/i386-linux-gnu/libcuda.so.340.107
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so.340.107
/usr/lib/x86_64-linux-gnu/libicudata.so.60
/usr/lib/x86_64-linux-gnu/libicudata.so.60.2
/usr/share/doc/libcuda1-340
/usr/share/doc/libcuda1-340/changelog.Debian.gz
/usr/share/doc/libcuda1-340/copyright
/usr/share/man/man1/nvidia-cuda-mps-control.1.gz
/usr/share/vim/vim80/indent/cuda.vim
/usr/share/vim/vim80/syntax/cuda.vim
/usr/src/linux-headers-4.15.0-66/include/linux/cuda.h
/usr/src/linux-headers-4.15.0-66/include/uapi/linux/cuda.h
/usr/src/linux-headers-4.15.0-70/include/linux/cuda.h
/usr/src/linux-headers-4.15.0-70/include/uapi/linux/cuda.h
/usr/src/linux-headers-4.15.0-72/include/linux/cuda.h
/usr/src/linux-headers-4.15.0-72/include/uapi/linux/cuda.h
/var/cache/apt/archives/libcuda1-340_340.107-0ubuntu0.18.04.4_amd64.deb
/var/lib/dpkg/info/libcuda1-340.list
/var/lib/dpkg/info/libcuda1-340.md5sums
/var/lib/dpkg/info/libcuda1-340.shlibs
/var/lib/dpkg/info/libcuda1-340.triggers
is this enough? and in case is not do i have to uninstall face/dlib/blas before repairing?
3. detect getting stuck at downloading files after ZM restarts - no idea. Haven't seen that happen before. Your logs show "Connection refused" when detect tries to download an image.
Yes this is really strange, i tried to open the url as soon as possible… perhaps i should try to force to reprocess the same event after having opened the url. 'll make some test with manual mode and see if i get the same errors...
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by asker »

fulvioth wrote: Thu Dec 19, 2019 9:15 pm
is this enough? and in case is not do i have to uninstall face/dlib/blas before repairing?

No ,at the end of the day you need to see if dlib detected CUDA and is using it. See https://gist.github.com/pliablepixels/f ... ou-need-it
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by asker »

fulvioth wrote: Thu Dec 19, 2019 9:15 pm
is this enough? and in case is not do i have to uninstall face/dlib/blas before repairing?

No ,at the end of the day you need to see if dlib detected CUDA and is using it. See https://gist.github.com/pliablepixels/f ... 4e25088af4

If you don't see

Code: Select all

>>> import dlib
>>> dlib.DLIB_USE_CUDA
True
>>> print (dlib.cuda.get_num_devices())
1
Then dlib is not using CUDA
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
fulvioth
Posts: 21
Joined: Wed Dec 18, 2019 5:27 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by fulvioth »

ok asker, i'm trying to repair things thanks
by the way… about the libs:

Code: Select all

sudo apt-get install build-essential cmake libjpeg-dev libpng-dev libtiff-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libgtk-3-dev libatlas-base-dev gfortran python3-dev pkg-config unzip ffmpeg qtbase5-dev  python3-dev python3-numpy libhdf5-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libtiff5-dev libtesseract-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev libpng16-16 libpng-dev libv4l-dev libtbb-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev v4l-utils libleptonica-dev
can you tell me the essential ones for my system?

EDIT:
i've made some progress, i found out that when dlib installed it was not going to use cuda because cuDNN was not found. So i installed cuDNN-dev but when i use:

Code: Select all

python ./setup.py install

running install
running bdist_egg
running egg_info
writing dlib.egg-info/PKG-INFO
writing dependency_links to dlib.egg-info/dependency_links.txt
writing top-level names to dlib.egg-info/top_level.txt
package init file 'dlib/__init__.py' not found (or not a regular file)
reading manifest file 'dlib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'tools/python/build*'
writing manifest file 'dlib.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Building extension for Python 3.6.9 (default, Nov  7 2019, 10:44:02)
Invoking CMake setup: 'cmake /home/fth/dlib/tools/python -DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/fth/dlib/build/lib.linux-x86_64-3.6 -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCMAKE_BUILD_TYPE=Release'
-- pybind11 v2.2.2
-- Using CMake version: 3.10.2
-- Compiling dlib version: 19.19.99
-- SSE4 instructions can be executed by the host processor.
-- Enabling SSE4 instructions
-- Found system copy of libpng: /usr/lib/x86_64-linux-gnu/libpng.so;/usr/lib/x86_64-linux-gnu/libz.so
-- Found system copy of libjpeg: /usr/lib/x86_64-linux-gnu/libjpeg.so
-- Searching for BLAS and LAPACK
-- Searching for BLAS and LAPACK
-- Checking for module 'cblas'
--   No package 'cblas' found
-- Found OpenBLAS library
-- Using OpenBLAS's built in LAPACK
-- C++11 activated.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/fth/dlib/build/temp.linux-x86_64-3.6
Invoking CMake build: 'cmake --build . --config Release -- -j3'
[ 68%] Built target dlib
[100%] Built target dlib_python
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/dlib.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for dlib.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/dlib.py to dlib.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying dlib.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlib.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlib.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlib.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlib.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
creating 'dist/dlib-19.19.99-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dlib-19.19.99-py3.6-linux-x86_64.egg
removing '/usr/local/lib/python3.6/dist-packages/dlib-19.19.99-py3.6-linux-x86_64.egg' (and everything under it)
creating /usr/local/lib/python3.6/dist-packages/dlib-19.19.99-py3.6-linux-x86_64.egg
Extracting dlib-19.19.99-py3.6-linux-x86_64.egg to /usr/local/lib/python3.6/dist-packages
dlib 19.19.99 is already the active version in easy-install.pth

Installed /usr/local/lib/python3.6/dist-packages/dlib-19.19.99-py3.6-linux-x86_64.egg
Processing dependencies for dlib==19.19.99
Finished processing dependencies for dlib==19.19.99
I guess it's not updating itself to use cuda because:

Code: Select all

>>> import dlib
>>> dlib.DLIB_USE_CUDA
False
>>> print (dlib.cuda.get_num_devices())
1
it still says false. How do i update this? if i change it within the shell (dlib.DLIB_USE_CUDA=True) it seems to lasts only for that session
fulvioth
Posts: 21
Joined: Wed Dec 18, 2019 5:27 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by fulvioth »

So i managed to have dlib use cuda but now i get an error when trying to enumerate cuda enabled device from python.
running devicequery:

Code: Select all

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL
So my latest nvidia driver are 340.107 (no newer available), cuda 10 requires >410, it looks that the latest cuda version supporting my gpu is 6.5 but when i install dlib it requires a cuda version >=7.5

in short… does this mean i can't use cuda with object detection?
User avatar
asker
Posts: 1553
Joined: Sun Mar 01, 2015 12:12 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by asker »

Maybe. If Dlib requires CUDA > 7.5 and you have 6.x, that is likely why it configures use cuda to false. Just setting that variable to true won't help.
I don't know much beyond the environment I have (which I posted in the gist)
I no longer work on zmNinja, zmeventnotification, pyzm or mlapi. I may respond on occasion based on my available time/interest.

Please read before posting:
How to set up logging properly
How to troubleshoot and report - ES
How to troubleshoot and report - zmNinja
ES docs
zmNinja docs
fulvioth
Posts: 21
Joined: Wed Dec 18, 2019 5:27 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by fulvioth »

Ok, i tried something else but still no luck. i guess i'll stick to blas until i can find a better gpu.
but still any chance to avoid at least the initial 3-4s for model initialization? it will still be an huge time saving for me (if it can be done only the first time).
fulvioth
Posts: 21
Joined: Wed Dec 18, 2019 5:27 pm

Re: Zoneminder and facedetection hook overall performance and detection issues

Post by fulvioth »

Hi, an update.
i managed to find another gpu GT730 and updated all the drivers.
Cuda works and recognize my GPU
dlib installed with cuda support
in python use cuda is true and enumerates 1 device.
last 3 face detection times (cnn):

Code: Select all

12/30/19 14:44:56 zmesdetect_m1[5626] DBG face.py:20 [Initializing face recognition with model:cnn upsample:1, jitters:1]
12/30/19 14:44:56 zmesdetect_m1[5626] DBG face.py:39 [pre-trained faces found, using that. If you want to add new images, remove: /var/lib/zmeventnotification/known_faces/faces.dat]
12/30/19 14:44:56 zmesdetect_m1[5626] DBG zm_detect.py:160 [|--> model:face init took: 2.958954s]
12/30/19 14:44:56 zmesdetect_m1[5626] DBG zm_detect.py:189 [Using model: face with /var/lib/zmeventnotification/images/1152-alarm.jpg]
12/30/19 14:45:04 zmesdetect_m1[5626] DBG zm_detect.py:194 [|--> model:face detection took: 7.85277s]

12/30/19 14:47:07 zmesdetect_m1[5678] DBG face.py:20 [Initializing face recognition with model:cnn upsample:1, jitters:1]
12/30/19 14:47:07 zmesdetect_m1[5678] DBG face.py:39 [pre-trained faces found, using that. If you want to add new images, remove: /var/lib/zmeventnotification/known_faces/faces.dat]
12/30/19 14:47:07 zmesdetect_m1[5678] DBG zm_detect.py:160 [|--> model:face init took: 3.198195s]
12/30/19 14:47:07 zmesdetect_m1[5678] DBG zm_detect.py:189 [Using model: face with /var/lib/zmeventnotification/images/1153-alarm.jpg]
12/30/19 14:47:10 zmesdetect_m1[5678] DBG zm_detect.py:194 [|--> model:face detection took: 3.059198s]

12/30/19 14:54:07 zmesdetect_m1[5742] DBG face.py:20 [Initializing face recognition with model:cnn upsample:1, jitters:1]
12/30/19 14:54:07 zmesdetect_m1[5742] DBG face.py:39 [pre-trained faces found, using that. If you want to add new images, remove: /var/lib/zmeventnotification/known_faces/faces.dat]
12/30/19 14:54:07 zmesdetect_m1[5742] DBG zm_detect.py:160 [|--> model:face init took: 2.985528s]
12/30/19 14:54:07 zmesdetect_m1[5742] DBG zm_detect.py:189 [Using model: face with /var/lib/zmeventnotification/images/1154-alarm.jpg]
12/30/19 14:54:10 zmesdetect_m1[5742] DBG zm_detect.py:194 [|--> model:face detection took: 3.108403s]
still high?
following your guide all seems ok, but i'm not sure if i need opencv 4 too for face detection.
any advice?
Post Reply