zmeventnotification stopped working

[Helipil0t]

I'm not sure what's going on here but seemingly overnight my zoneminder setup no longer detects object / faces. What's more, every once in a while the server will crash and all streams will drop. zmc process is killed and drop off the GPU.

I'm running 1.36.32. I tried installing the latest version of zmeventnotifications. Testing an event ID I get the following:

Code: Select all

sudo -u www-data /var/lib/zmeventnotification/bin/zm_detect.py --config /etc/zm/objectconfig.ini  --eventid 26449 --monitorid 4 --debug
11/27/22 09:26:19 zmesdetect_m4[44627] INF ZMLog.py:292 [Setting up signal handler for logs]

11/27/22 09:26:19 zmesdetect_m4[44627] INF ZMLog.py:301 [Switching global logger to ZMLog]

11/27/22 09:26:19 zmesdetect_m4[44627] INF zm_detect.py:284 [---------| app:6.1.28, pyzm:0.3.56, ES:6.1.28
 , OpenCV:4.5.3|------------]

11/27/22 09:26:19 zmesdetect_m4[44627] INF utils.py:404 [Reading config from: /etc/zm/objectconfig.ini]

11/27/22 09:26:19 zmesdetect_m4[44627] INF utils.py:409 [Reading secrets from: /etc/zm/secrets.ini]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_PORTAL]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_USER]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_PASSWORD]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ZM_API_PORTAL]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ML_USER]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !ML_PASSWORD]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:381 [Secret token found in config: !PLATEREC_ALPR_KEY]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG1 utils.py:444 [allowing self-signed certs to work...]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG2 utils.py:454 [Now checking for monitor overrides]

11/27/22 09:26:19 zmesdetect_m4[44627] DBG3 utils.py:521 [Finally, doing parameter substitution]

11/27/22 09:26:19 zmesdetect_m4[44627] INF zm_detect.py:309 [Importing local classes for Object/Face]

11/27/22 09:26:20 zmesdetect_m4[44627] INF zm_detect.py:334 [Connecting with ZM APIs]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 api.py:72 [API SSL certificate check has been disbled]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 api.py:181 [using username/password for login]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 api.py:210 [Using new token API]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 api.py:217 [Access token expires on:2022-11-27 11:26:20.365089 [7200s]]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 api.py:221 [Refresh token expires on:2022-11-28 09:26:20.365727 [86400s]]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 zm_detect.py:342 [using ml_sequence]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 zm_detect.py:354 [using stream_sequence]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:160 [Resetting models, will be loaded on next run]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 detect_sequence.py:634 [Using automatic locking as we are switching between models]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:51 [Media get SSL certificate check has been disbled]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:99 [Using URL 26449 for stream]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:114 [We will only process frames: ['snapshot', 'alarm']]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 Media.py:137 [No need to start streams, we are picking images from https://seanstech.ca:64443/zm/index.php?view=image&eid=26449]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 Media.py:271 [Reading https://seanstech.ca:64443/zm/index.php?view=image&eid=26449&fid=snapshot]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 api.py:144 [No need to relogin as access token still has 119.99981745 minutes remaining]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 api.py:272 [make_request called with url=https://seanstech.ca:64443/zm/index.php?view=image&eid=26449&fid=snapshot payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNjY5NTU5MTgwLCJleHAiOjE2Njk1NjYzODAsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.ZOOUbYGmc9fWPJ7xM9bJ_it9BIuOR0YTxEhAM9W92S4'}]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:654 [perf: Starting for frame:snapshot]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:664 [Sequence of detection types to execute: ['object', 'face', 'alpr']]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:669 [============ Frame: snapshot Running object detection type in sequence ==================]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 detect_sequence.py:174 [Skipping TPU object detection as it is disabled]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 detect_sequence.py:178 [Loading sequence: YoloV4 GPU/CPU]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 detect_sequence.py:179 [Initializing model  type:object with options:{'name': 'YoloV4 GPU/CPU', 'enabled': 'yes', 'object_config': '/var/lib/zmeventnotification/models/yolov4/yolov4.cfg', 'object_weights': '/var/lib/zmeventnotification/models/yolov4/yolov4.weights', 'object_labels': '/var/lib/zmeventnotification/models/yolov4/coco.names', 'object_min_confidence': 0.3, 'object_framework': 'opencv', 'object_processor': 'gpu', 'gpu_max_processes': 1, 'gpu_max_lock_wait': 100, 'cpu_max_processes': 3, 'cpu_max_lock_wait': 100, 'max_detection_size': '90%', 'disable_locks': 'no'}]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:38 [portalock: max:1, name:pyzm_uid33_gpu_lock, timeout:100]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 detect_sequence.py:689 [object has a same_model_sequence strategy of first]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 detect_sequence.py:701 [--------- Frame:snapshot Running variation: #1 -------------]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:136 [detect extracted image dimensions as: 800wx1066h]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:50 [Waiting for pyzm_uid33_gpu_lock portalock...]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:52 [Got pyzm_uid33_gpu_lock portalock]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 yolo.py:85 [|--------- Loading "YoloV4 GPU/CPU" model from disk -------------|]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 yolo.py:100 [perf: processor:gpu Yolo initialization (loading /var/lib/zmeventnotification/models/yolov4/yolov4.weights model from disk) took: 250.32 ms]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG2 yolo.py:115 [Setting CUDA backend for OpenCV]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG3 yolo.py:116 [If you did not set your CUDA_ARCH_BIN correctly during OpenCV compilation, you will get errors during detection related to invalid device/make_policy]

11/27/22 09:26:20 zmesdetect_m4[44627] DBG1 yolo.py:160 [|---------- YOLO (input image: 800w*1066h, model resize dimensions: 416w*416h) ----------|]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 yolo.py:68 [Released pyzm_uid33_gpu_lock portalock]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG1 yolo.py:185 [perf: processor:gpu Yolo detection took: 9649.06 ms]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 yolo.py:220 [perf: processor:gpu Yolo NMS filtering took: 4.14 ms]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:66 [core model detection over, got 4 objects. Now filtering]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:70 [Max object size found to be: 90%]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:78 [Converted 90% to 767520.0]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:101 [Ignoring car [520, 564, 592, 618] as conf. level 0.28317883610725403 is lower than 0.3]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:93 [Ignoring object:aeroplane, as it's area: 871700px exceeds max_object_area of 767520.0px]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 object.py:103 [Returning filtered list of 2 objects.]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:711 [This model iteration inside object found: labels: ['person', 'suitcase'],conf:[0.9284842014312744, 0.6206374168395996]]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:424 [Max object size found to be: 90%]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:432 [Converted 90% to 767520.0]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:448 [No polygons, adding full image polygon: {'name': 'full_image', 'value': [(0, 0), (1536, 0), (1536, 2048), (0, 2048)], 'pattern': None}]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG3 detect_sequence.py:228 [resized polygons x=0.5208333333333334/y=0.5205078125: [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 1066), (0, 1066)], 'pattern': None}]]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:502 [intersection: object:person,POLYGON ((526 426, 796 426, 796 1006, 526 1006, 526 426)) intersects polygon:full_image,POLYGON ((0 0, 800 0, 800 1066, 0 1066, 0 0))]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:518 [Using global match pattern: (person|car|motorbike|bus|truck|boat)]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:526 [full_image intersects object:person[[(526, 426), (796, 426), (796, 1006), (526, 1006)]]]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:502 [intersection: object:suitcase,POLYGON ((365 863, 631 863, 631 1049, 365 1049, 365 863)) intersects polygon:full_image,POLYGON ((0 0, 800 0, 800 1066, 0 1066, 0 0))]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:518 [Using global match pattern: (person|car|motorbike|bus|truck|boat)]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:535 [full_image intersects object:suitcase[[(365, 863), (631, 863), (631, 1049), (365, 1049)]] but does NOT match your detect pattern filter]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:753 [breaking out of same model loop, as matches found and strategy is "first"]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG1 detect_sequence.py:669 [============ Frame: snapshot Running face detection type in sequence ==================]

11/27/22 09:26:30 zmesdetect_m4[44627] DBG2 detect_sequence.py:190 [Skipping TPU face detection as it is disabled]

Illegal instruction

Clearly it detects the object. But then stops with "Illegal instruction"

When the server crashes I have to restart the zoneminder service in order to get the monitors back online until the next detection runs.. Then it may or may not crash again.
How can I further diagnose this issue?

Thanks!

[Helipil0t]

Quick update.. The crash I believe is as a result of running out of ram. dmesg shows:

Code: Select all

Out of memory: Killed process 7665 (zmc) total-vm:8454820kB, anon-rss:488988kB, file-rss:132124kB, shmem-rss:488056kB, UID:33 pgtables:5412kB oom_score_adj:0

I'll up my memory a bit. Should I set "Maximum Image Buffer Size (frames):" ?

[Magic919]

I think it’s better to constrain it. I never get oom killed.

[Helipil0t]

K so that sorts out the crashes. Still can't get zmeventnotification working properly. No idea what's going on with it. The debug doesn't share too many clues.

[Magic919]

ZMES is clearly finding objects.

I’d suggest running the instance on the commandline to see the debug. Then manually running the check above and the others that are suggested in the docs. Something must show up.

[Helipil0t]

Ok so I think there's issues with yolo.py:

Code: Select all

11/28/22 01:35:44 zmesdetect_m3[1310] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/object.py", line 65, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 168, in detect
    ln = self.get_output_layers()
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 129, in get_output_layers
    output_layers = [
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 130, in <listcomp>
    layer_names[i[0] - 1] for i in self.net.getUnconnectedOutLayers()
IndexError: invalid index to scalar variable.

This is running the latest zmeventnotification from github. So the "hotfix" no longer works. Running OpenCV 4.6.0-dev, Cuda 11.8

[Magic919]

Ok.

[Helipil0t]

So I thought this code was fixed a while ago with the following:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        if self.is_get_unconnected_api_list:
            output_layers = [
                layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        else:
            output_layers = [
                layer_names[i[0] - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        return output_layers

I removed the if else statement and just used:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        output_layers = [
            layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
        ]
        return output_layers

Run again and get the following error:

Code: Select all

11/28/22 10:41:10 zmesdetect_m3[4068] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/object.py", line 65, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 225, in detect
    i = i[0]
IndexError: invalid index to scalar variable.

I have Python 3.10.6.

Maybe I should use an older version of Python?

[Helipil0t]

Update:

I commented out the if statement in the following:

Code: Select all

        # now filter out with configured yolo confidence, so we can see rejections in log
        for i in indices:
           # if not self.is_get_unconnected_api_list:
           #     i = i[0]
            box = boxes[i]
            x = box[0]
            y = box[1]
            w = box[2]
            h = box[3]

And everything seems to be working again.
Wish I knew more about what I was doing. Is it the python interpreter not handing indices the same way?

[Magic919]

I've no idea.

There is some mention of the topic by Asker. He understands it

viewtopic.php?t=31129

[lorenjz]

I'm having the same issue however, commenting out the if statement mentioned here

Update:

I commented out the if statement in the following:

Code: Select all

        # now filter out with configured yolo confidence, so we can see rejections in log
        for i in indices:
           # if not self.is_get_unconnected_api_list:
           #     i = i[0]
            box = boxes[i]
            x = box[0]
            y = box[1]
            w = box[2]
            h = box[3]

And everything seems to be working again.
Wish I knew more about what I was doing. Is it the python interpreter not handing indices the same way?

or the else if mentioned here:

So I thought this code was fixed a while ago with the following:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        if self.is_get_unconnected_api_list:
            output_layers = [
                layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        else:
            output_layers = [
                layer_names[i[0] - 1] for i in self.net.getUnconnectedOutLayers()
            ]
        return output_layers

I removed the if else statement and just used:

Code: Select all

    def get_output_layers(self):
        layer_names = self.net.getLayerNames()
        output_layers = [
            layer_names[i - 1] for i in self.net.getUnconnectedOutLayers()
        ]
        return output_layers

Run again and get the following error:

Code: Select all

11/28/22 10:41:10 zmesdetect_m3[4068] DBG2 detect_sequence.py:714 [Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/detect_sequence.py", line 710, in detect_stream
    _b,_l,_c,_m = m.detect(image=frame)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/object.py", line 65, in detect
    b,l,c,_model_names = self.model.detect(image)
  File "/usr/local/lib/python3.10/dist-packages/pyzm/ml/yolo.py", line 225, in detect
    i = i[0]
IndexError: invalid index to scalar variable.

I have Python 3.10.6.

Maybe I should use an older version of Python?

change the error but no matter the combination of commented or not for either or both don't allow a successful analysis. Following along for any further developments....

[Magic919]

You might get help in the Slack channel.

[volfan6415]

Is there a fix for this? I am having the same issue.

[eresonance]

I have python3.11 and OpenCV 4.7.0 and I see the same issue. I think there's a problem detecting the opencv version.

Open up the yolo.py script:

Code: Select all

sudo vim /usr/local/lib/python3.11/site-packages/pyzm/ml/yolo.py

Go to line 95:

Code: Select all

        (maj, minor, patch) = cv2.__version__.split('.')
        min_ver = int(maj + minor)
        patch = int(patch) if patch.isdigit() else 0
        if min_ver >= 45 and patch >=4:
            # see https://github.com/opencv/opencv/issues/20923
            # we need to modify Yolo code not to expect a nested structure
            g.logger.Debug(1, 'You are using OpenCV >= 4.5.4, making sure we fix getUnconnectedOutLayers() API')
            self.is_get_unconnected_api_list = True

We want that to look like:

Code: Select all

        (maj, minor, patch) = cv2.__version__.split('.')
        min_ver = int(maj + minor + patch)
        if min_ver >= 454:
            # see https://github.com/opencv/opencv/issues/20923
            # we need to modify Yolo code not to expect a nested structure
            g.logger.Debug(1, 'You are using OpenCV >= 4.5.4, making sure we fix getUnconnectedOutLayers() API')
            self.is_get_unconnected_api_list = True

That will handle the case where your minor version is > 5 but your patch is < 4.

[Pedulla]

On a 1.36.33 ZM server...

04/30/23 23:52:47 zmesdetect_m10[881377] ERR detect_sequence.py:713 [Error running model: invalid index to scalar variable.]

using

Code: Select all

>>> from cv2 import __version__
>>> __version__
'4.7.0'

Is this something else?

New install of ES, wiped out the models and reinstalled just in case and fix above runs, but does not solve.