zmeventnotification hook download fail

[h1pp13p373]

Greetings!

I'm working on setting up zmeventnotification with hooks and am struggling with a download url error. The event server seems to be working ok, but I can't get a hook to work due to a download error. I am using SSL, but I do not have authentication set up for the zoneminder portal as it's locked away on my LAN.

I'm on kubuntu 20.04 with zoneminder 1.34.16. Here's my zneventnotification start:

Code: Select all

$ sudo -u www-data /usr/bin/zmeventnotification.pl
06/27/2020 12:09:16.026407 zmeventnotification[483655].INF [main:881] [PARENT: using config file: /etc/zm/zmeventnotification.ini]
06/27/2020 12:09:16.048231 zmeventnotification[483655].INF [main:881] [PARENT: using secrets file: /etc/zm/secrets.ini]
06/27/2020 12:09:16.230741 zmeventnotification[483655].INF [main:881] [PARENT: Push enabled via FCM]
06/27/2020 12:09:16.240616 zmeventnotification[483655].INF [main:881] [PARENT: MQTT Disabled]
06/27/2020 12:09:16.250109 zmeventnotification[483655].INF [main:881] [PARENT: |------- Starting ES version: 5.13 ---------|]
Can't ignore signal CHLD, forcing to default.
06/27/2020 12:09:16.445339 zmeventnotification[483655].INF [main:881] [PARENT: Event Notification daemon v 5.13 starting]
06/27/2020 12:09:16.457578 zmeventnotification[483655].INF [main:881] [PARENT: Re-loading monitors]
06/27/2020 12:09:16.489880 zmeventnotification[483655].INF [main:881] [PARENT: Secure WS(WSS) is enabled...]
06/27/2020 12:09:16.502050 zmeventnotification[483655].INF [main:881] [PARENT: Web Socket Event Server listening on port 9000]

When I test a hook, it fails. I've tried various versions of the ZMES_PICTURE_URL variable in /etc/zm/secrets.ini and it doesn't seem to be using them. I even changed to an erroneous web address and it still showed "Trying to download /index.php?view=image&eid=2942&fid=alarm".

Code: Select all

$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 2942 garage
^ returns without info

log:
==> /var/log/zm/zmesdetect_mgarage.log <==
06/27/20 12:45:11 zmesdetect_mgarage[485065] INF zm_detect.py:177 [---------| hook version: 5.13.3, ES version: 5.13 , OpenCV version: 4.4.0-pre|------------]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG1 utils.py:265 [secret filename: /etc/zm/secrets.ini]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:238 [Secret token found in config: !ZM_API_PORTAL]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:238 [Secret token found in config: !ML_USER]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:238 [Secret token found in config: !ML_PASSWORD]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:238 [Secret token found in config: !PLATEREC_ALPR_KEY]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG1 utils.py:288 [allowing self-signed certs to work...]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [config] is '/var/lib/zmeventnotification/models/yolov3/yolov3.cfg' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [weights] is '/var/lib/zmeventnotification/models/yolov3/yolov3.weights' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [labels] is '/var/lib/zmeventnotification/models/yolov3/yolov3_classes.txt' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [tiny_config] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.cfg' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [tiny_weights] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.weights' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [tiny_labels] is '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.txt' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [known_images_path] is '/var/lib/zmeventnotification/known_faces' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG2 utils.py:342 [key [unknown_images_path] is '/var/lib/zmeventnotification/unknown_faces' after substitution]
06/27/20 12:45:11 zmesdetect_mgarage[485065] INF zm_detect.py:206 [Importing local classes for Yolo/Face]
06/27/20 12:45:11 zmesdetect_mgarage[485065] DBG1 utils.py:143 [Trying to download /index.php?view=image&eid=2942&fid=alarm]
06/27/20 12:45:11 zmesdetect_mgarage[485065] ERR zm_detect.py:220 [Error downloading files: unknown url type: '/index.php?view=image&eid=2942&fid=alarm']
06/27/20 12:45:11 zmesdetect_mgarage[485065] FAT zm_detect.py:221 [animation: Traceback:Traceback (most recent call last):
  File "/var/lib/zmeventnotification/bin/zm_detect.py", line 217, in <module>
    filename1, filename2, filename1_bbox, filename2_bbox = utils.download_files(
  File "/usr/local/lib/python3.8/dist-packages/zmes_hook_helpers/utils.py", line 145, in download_files
    input_file = opener.open(url)
  File "/usr/lib/python3.8/urllib/request.py", line 509, in open
    req = Request(fullurl, data)
  File "/usr/lib/python3.8/urllib/request.py", line 328, in __init__
    self.full_url = url
  File "/usr/lib/python3.8/urllib/request.py", line 354, in full_url
    self._parse()
  File "/usr/lib/python3.8/urllib/request.py", line 383, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: '/index.php?view=image&eid=2942&fid=alarm'

Currently, I have the following in /etc/zm/secrets.ini:

Code: Select all

ZMES_PICTURE_URL=https://margaritaville.lucky.org/zm/?view=image&eid=EVENTID&fid=snapshot&width=600

and I have tested browsing to that url (and substituting EVENTID with a real event id) and the image loads without issue. Here's my config:

Code: Select all

$ sudo /usr/bin/zmeventnotification.pl --check-config
06/27/2020 12:53:00.541012 zmeventnotification[485225].INF [main:881] [PARENT: using config file: /etc/zm/zmeventnotification.ini]
06/27/2020 12:53:00.562456 zmeventnotification[485225].INF [main:881] [PARENT: using secrets file: /etc/zm/secrets.ini]
  
Configuration (read /etc/zm/zmeventnotification.ini):
  
Secrets file.......................... /etc/zm/secrets.ini
Base data path........................ /var/lib/zmeventnotification
Restart interval (secs)............... (undefined)
  
Use admin interface .................. no
Admin interface password.............. (undefined)
Admin interface persistence file ..... /var/lib/zmeventnotification/misc/escontrol_interface.dat
  
Port ................................. 9000
Address .............................. [::]
Event check interval ................. 5
Monitor reload interval .............. 300
Skipped monitors...................... (undefined)
  
Auth enabled ......................... yes
Auth timeout ......................... 20
  
Use API Push.......................... no
API Push Script....................... (undefined)
  
Use FCM .............................. yes
FCM Date Format....................... %I:%M %p, %d-%b
FCM API key .......................... (defined)
Token file ........................... /var/lib/zmeventnotification/push/tokens.txt

Use MQTT ............................. no
MQTT Server .......................... 127.0.0.1
MQTT Username ........................ (undefined)
MQTT Password ........................ (undefined)
MQTT Retain .......................... no
MQTT Tick Interval ................... 15
  
SSL enabled .......................... yes
SSL cert file ........................ /etc/ssl/certs/apache-selfsigned.crt
SSL key file ......................... /etc/ssl/private/apache-selfsigned.key
  
Verbose .............................. no
ES Debug level.........................2
Read alarm cause ..................... yes
Tag alarm event id ................... yes
Use custom notification sound ........ no
Send event end notification............yes
  
Use Hooks............................. yes
Hook Script on Event Start ........... '/var/lib/zmeventnotification/bin/zm_event_start.sh'
Hook Script on Event End.............. '/var/lib/zmeventnotification/bin/zm_event_end.sh'
Hook Skipped monitors................. (undefined)
  
Notify on Event Start (hook success).. all
Notify on Event Start (hook fail)..... none
Notify on Event End (hook success).... fcm,web,api
Notify on Event End (hook fail)....... none
Notify End only if Start success...... yes
  
Use Hook Description.................. yes
Keep frame match type................. yes
Store Frame in ZM......................yes
  
Picture URL .......................... https://margaritaville.lucky.org/zm/?view=image&eid=EVENTID&fid=snapshot&width=600
Include picture....................... yes
Picture username ..................... zmuser
Picture password ..................... (defined)

I'd appreciate any advise, and thanks!

[asker]

Code: Select all

06/27/20 12:45:11 zmesdetect_mgarage[485065] ERR zm_detect.py:220 [Error downloading files: unknown url type: '/index.php?view=image&eid=2942&fid=alarm']

This means you haven't defined ZM_PORTAL in /etc/zm/secrets.ini.

ZMES_PICTURE_URL is only used for images in push notifications.

[h1pp13p373]

That makes sense, thanks! I have ZM_PORTAL set like this now:

Code: Select all

ZM_PORTAL=https://margaritaville.lucky.org/zm

I also tried setting it the same as ZMES_PICTURE_URL, but I get the same error. How should that be set? Here's my redacted secrets.ini:

Code: Select all

# cat secrets.ini
# your secrets file
[secrets]

# fid can have the following values:
# a particular <frameid>, alarm or snapshot
# starting ZM 1.35, you can also specify
# objdetect_mp4, objdetect_gif or objdetect_image
# this needs create_animation enabled in objectconfig.ini and associated flags
ZMES_PICTURE_URL=https://margaritaville.lucky.org/zm/?view=image&eid=EVENTID&fid=snapshot&width=600
ZM_USER=zmuser
ZM_PASSWORD=password
#ES_ADMIN_INTERFACE_PASSWORD=your_admin_interface_password

ZM_PORTAL=https://margaritaville.lucky.org/zm
ZM_API_PORTAL=https://margaritaville.lucky.org/zm/api
ES_CERT_FILE=/etc/ssl/certs/apache-selfsigned.crt
ES_KEY_FILE=/etc/ssl/private/apache-selfsigned.key
ML_USER=your_mlapi_user
ML_PASSWORD=your_mlapi_password
PLATEREC_ALPR_KEY=your_plate_recognizer_api_key
OPENALPR_ALPR_KEY=your_openalpr_api_key

ESCONTROL_INTERFACE_PASSWORD=yourescontrolpassword

MQTT_USERNAME=your_mqtt_username
MQTT_PASSWORD=your_mqtt_password

PUSHOVER_APP_TOKEN=your_pushover_app_token
PUSHOVER_USER_KEY=your_pushover_user_key

[SkippyDo]

Kind of sounds like my "world" (my thread: viewtopic.php?f=33&t=29595)

[asker]

No. Unrelated to your issue if you see his logs.

[SkippyDo]

Shouldn't the error show up with something different now that ZM_PORTAL is defined?

Make sure file access permissions are correct for the web user you're running as.

I was under the impression that using self-signed certs wasn't permissible (for, at least, getting notifications via FCM [maybe just picture issue?]).

[asker]

OP:

- Upgrade. Looks like you are running 5.13.3. Latest is 5.15.5
- Restart ZMES
- Post debug logs of zm_detect.py from the start again.
- Forget about the picture url for now
- Are you sure your user is "zmuser"? That is usually the DB credentials. You need to put in your web console credentials here (typically admin/<password>). Read the comments above each attribute to make sure you are following the intent of each attribute.
- In you secrets.ini, you have a whole bunch of attributes uncommented with default settings - if you don't use them, comment them out or you will face issues.

[h1pp13p373]

Thanks again, asker! I'm working on the upgrade and will fix up secrets.ini, commenting out those lines.

I do have the 'real' credentials in my file, but I was using the database credentials. I don't have authentication set up for zm, but it sounds like I need to for this to work, so I'll set that up as well, then enter the web ui credentials in secrets.ini.

I'll update once I've made the changes and tested again.

Happy Sunday!

[h1pp13p373]

WOOHOO!!! Thanks so much asker! It was a combo of all you said, plus a few things I needed to comment out from objectconfig.ini that I'm not usinmg yet. I fixed up files, upgraded, and BAM.

Code: Select all

$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 3041 frontdoor
[a] detected:truck:82% --SPLIT--[{"type": "object", "label": "truck", "box": [842, 230, 1040, 362], "confidence": "81.80%"}]

Awesome sauce. Now it's on to the fun stuff like alpr and tweaking... this is way cool, thanks again!!

[h1pp13p373]

I hit another little hurdle. Detection seems to be working, but it's looking for a 'faces.dat' file that isn't on my system. Is this something I need to generate or download? I do have the directory, '/var/lib/zmeventnotification/known_faces', just not faces.dat.

Code: Select all

$ sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 4732 frontdoor
[a] detected:person:96% truck:84% --SPLIT--[{"type": "object", "label": "person", "box": [798, 238, 842, 350], "confidence": "95.90%"}, {"type": "object", "label": "truck", "box": [676, 182, 810, 320], "confidence": "84.30%"}, {"type": "object", "label": "truck", "box": [845, 231, 1035, 359], "confidence": "65.57%"}]

and log:
06/28/20 12:29:01 zmesdetect_mfrontdoor[543731] INF zm_detect.py:181 [---------| hook version: 5.15.5, ES version: 5.15 , OpenCV version: 4.4.0-pre|------------]
06/28/20 12:29:01 zmesdetect_mfrontdoor[543731] INF zm_detect.py:210 [Importing local classes for Yolo/Face]
06/28/20 12:29:05 zmesdetect_mfrontdoor[543731] INF yolo.py:159 [object:person at [798, 238, 842, 350] has a acceptable confidence:0.9590409994125366 compared to min confidence of: 0.3, add
ing]
06/28/20 12:29:05 zmesdetect_mfrontdoor[543731] INF yolo.py:159 [object:truck at [676, 182, 810, 320] has a acceptable confidence:0.8429875373840332 compared to min confidence of: 0.3, addi
ng]
06/28/20 12:29:05 zmesdetect_mfrontdoor[543731] INF yolo.py:159 [object:truck at [845, 231, 1035, 359] has a acceptable confidence:0.6556857228279114 compared to min confidence of: 0.3, add
ing]
06/28/20 12:29:05 zmesdetect_mfrontdoor[543731] INF zm_detect.py:619 [labels found: ['person', 'truck', 'truck']]
06/28/20 12:29:10 zmesdetect_mfrontdoor[543731] ERR face_train.py:115 [No known faces found to train, encoding file not created]
06/28/20 12:29:10 zmesdetect_mfrontdoor[543731] ERR face.py:64 [Error loading KNN model: [Errno 2] No such file or directory: '/var/lib/zmeventnotification/known_faces/faces.dat']
06/28/20 12:29:11 zmesdetect_mfrontdoor[543731] INF zm_detect.py:660 [Removing matches to past detections]
06/28/20 12:29:11 zmesdetect_mfrontdoor[543731] ERR zm_detect.py:728 [Could not write image to ZoneMinder as eventpath not present]
06/28/20 12:29:11 zmesdetect_mfrontdoor[543731] INF zm_detect.py:761 [Prediction string:[a] detected:person:96% truck:84% ]

I'm close... I can feel it.

[asker]

Glad you got it working.
A missing faces.dat means you have face recognition enabled but no trained faces. Either turn off face recognition or train faces as the docs describe.

[h1pp13p373]

Gotcha, that makes sense. Much appreciated!! This is really cool.

[isisyodin]

Hi h1pp13p373,

I am having a hard time getting hooks to work which may be similar to what you were going through. I am unsure about my configuration files: secrets.ini, objectconfig.ini, and zmeventnotification.ini. Do you mind sharing your files here?
I am able to retrieve an image by using my zm portal variable and eid without any issues. Also notifications work fine.