I am working on a new config system to make things a lot easier to work with. The thing about it is that it will move the config files (objectconfig and mlapiconfig) to YAML. Here is some testing output so far ->
Code: Select all
/home/tyler/PycharmProjects/my_mlapi/venv/bin/python /home/tyler/PycharmProjects/my_mlapi/pyzm/helpers/new_yaml.py
config:init: confirming config file exists and is a file -> '/shared/objectconfig.yml'
config:hash: the SHA-256 hexdigest for config file '/shared/objectconfig.yml' -> 1008afdb73f5ded9d6b147fa0b664d089cf1877b0b08ce699157857c985ba91d
config:zmes: added 27 default keys to the base config (these keys were missing) -> force_mpd, same_model_high_conf, skip_mons, basic_user, basic_password, max_detection_size, contained_area, hass_enabled, hass_notify, hass_cooldown, push_jpg, push_jpg_key, push_gif, push_gif_key, push_debug_device, push_cooldown, mqtt_topic, mqtt_port, tls_cert, tls_key, ml_enable, alpr_url, platerec_regions, openalpr_recognize_vehicle, openalpr_country, openalpr_state, openalpr_min_confidence!
config:zmes: confirming the configured secrets file exists and is a file -> '/shared/zm_secrets.yml'
config:zmes: secrets file confirmed! starting {[secrets]} substitution
config:zmes: successfully replaced 21 secrets in the base config -> AWS_SECRET_ACCESS_KEY, ZM_PORTAL, AWS_ACCESS_KEY_ID, MQTT_PASSWORD, ML_USER, HA_SERVER, HA_TOKEN, PUSHOVER_USER_KEY, ZM_PASSWORD, ZM_API_PORTAL, PUSHOVER_USER, PUSHOVER_PASS, ZM_USER, PLATEREC_ALPR_KEY, ZM_ML_USER, PUSHOVER_APP_TOKEN, mlapi_one_key, ZM_ML_PASSWORD, ML_PASSWORD, PUSHOVER_ERR_TOKEN, MQTT_USERNAME
config:zmes: substituting vars for the base config keys
config:zmes: successfully replaced 1 default sub var in the base config -> 'base_data_path'
config:zmes: starting {{var}} substitution
config:zmes: successfully replaced 51 sub vars in the base config -> platerec_stats, yolo4_object_framework, alpr_key, face_train_model, yolo4_object_weights, known_images_path, tpu_object_weights_mobilenetv2, unknown_images_path, save_unknown_faces, frame_strategy, same_model_sequence_strategy, face_detection_pattern, past_det_max_diff_area, platerec_min_score, gpu_max_processes, tpu_max_processes, object_min_confidence, yolo4_object_processor, model_sequence, tpu_face_weights_mobilenetv2, smart_fs_thresh, face_recog_dist_threshold, tpu_object_labels, max_detection_size, tpu_object_framework, match_past_detections, disable_locks, cpu_max_processes, openalpr_cmdline_min_confidence, fp16_target, contained_area, alpr_detection_pattern, face_num_jitters, tpu_object_weights_mobiledet, gpu_max_lock_wait, face_upsample_times, yolo4_object_config, frame_set, yolo4_object_labels, save_unknown_faces_leeway_pixels, resize, show_models, openalpr_cmdline_params, face_detection_framework, platerec_min_dscore, tpu_min_confidence, tpu_max_lock_wait, face_model, openalpr_cmdline_binary, cpu_max_lock_wait, object_detection_pattern
config:zmes: attempting to build an overrode config from monitor '1' overrides
config:zmes: monitor '1' overrides, starting {[secrets]} substitution
config:zmes: no secrets were replace during overrode config build
config:zmes: no vars were replace during overrode config build
config:zmes: found a polygon for defined zone 'front_yard' -> [(0, 877), (2170, 553), (3822, 1131), (3822, 2141), (0, 2159)]
config:zmes: detection pattern for monitor '1' defined zone 'front_yard' -> (person|dog|cat)
config:zmes: these are they keys that were overrode in the base config -> model_sequence, object_detection_pattern, frame_set, hass_notify, hass_cooldown
config:zmes: these are the keys from overrides that did not have a base value to override but are now in the config -> person_min_confidence, hass_person
ephemeral detection_patterns = {'front_yard': '(person|dog|cat)'}
self.polygons = {1: {'front_yard': [{'polygon': [(0, 877), (2170, 553), (3822, 1131), (3822, 2141), (0, 2159)], 'pattern': '(person|dog|cat)'}]}}
perf:config:zmes: 0.000497 seconds for monitor '1' overrode config build
config:zmes: attempting to build an overrode config from monitor '3' overrides
config:zmes: monitor '3' overrides, starting {[secrets]} substitution
config:zmes: no secrets were replace during overrode config build
config:zmes: no vars were replace during overrode config build
config:zmes: found a polygon for defined zone 'parking_pad' -> [(5, 408), (660, 175), (1695, 100), (1910, 275), (1910, 1070), (0, 1080)]
config:zmes: detection pattern for monitor '3' defined zone 'parking_pad' -> (person|dog|cat)
config:zmes: these are they keys that were overrode in the base config -> model_sequence, object_detection_pattern, frame_set, hass_notify, hass_cooldown
config:zmes: these are the keys from overrides that did not have a base value to override but are now in the config -> person_min_confidence
ephemeral detection_patterns = {'parking_pad': '(person|dog|cat)'}
self.polygons = {3: {'parking_pad': [{'polygon': [(5, 408), (660, 175), (1695, 100), (1910, 275), (1910, 1070), (0, 1080)], 'pattern': '(person|dog|cat)'}]}}
perf:config:zmes: 0.000253 seconds for monitor '3' overrode config build
config:zmes: attempting to build an overrode config from monitor '4' overrides
config:zmes: monitor '4' overrides, starting {[secrets]} substitution
config:zmes: no secrets were replace during overrode config build
config:zmes: no vars were replace during overrode config build
config:zmes: found a polygon for defined zone 'back_yard' -> [(0, 547), (824, 246), (1511, 161), (1706, 1070), (0, 1080)]
config:zmes: detection pattern for monitor '4' defined zone 'back_yard' -> (person)
config:zmes: these are they keys that were overrode in the base config -> model_sequence, object_detection_pattern, frame_set, hass_notify, hass_cooldown
config:zmes: these are the keys from overrides that did not have a base value to override but are now in the config -> person_min_confidence
ephemeral detection_patterns = {'back_yard': '(person)'}
self.polygons = {4: {'back_yard': [{'polygon': [(0, 547), (824, 246), (1511, 161), (1706, 1070), (0, 1080)], 'pattern': '(person)'}]}}
perf:config:zmes: 0.000228 seconds for monitor '4' overrode config build
config:zmes: attempting to build an overrode config from monitor '5' overrides
config:zmes: monitor '5' overrides, starting {[secrets]} substitution
config:zmes: no secrets were replace during overrode config build
config:zmes: no vars were replace during overrode config build
config:zmes: these are they keys that were overrode in the base config -> object_detection_pattern, model_sequence, push_cooldown, frame_strategy, frame_set, face_upsample_times, face_num_jitters
config:zmes: these are the keys from overrides that did not have a base value to override but are now in the config -> same_model_seq_strategy
ephemeral detection_patterns = {}
self.polygons = {}
perf:config:zmes: 0.000157 seconds for monitor '5' overrode config build
config:init: confirming config file exists and is a file -> '/shared/mlapiconfig.yml'
config:hash: the SHA-256 hexdigest for config file '/shared/mlapiconfig.yml' -> 78d495593def4be08bbf992e7b96ead8e625e2dc2a3e3e7c9aaf98985cbafbea
config:mlapi: added 20 default keys to the base config (these keys were missing) -> log_name, log_path, force_mpd, auth_enabled, api_portal, basic_auth_user, basic_auth_password, import_zm_zones, only_triggered_zm_zones, wait, max_detection_size, detection_sequence, detection_mode, object_framework, alpr_api_type, alpr_url, openalpr_recognize_vehicle, openalpr_country, openalpr_state, openalpr_min_confidence!
config:mlapi: confirming the configured secrets file exists and is a file -> '/shared/ml_secrets.yml'
config:mlapi: secrets file confirmed! starting {[secrets]} substitution
config:mlapi: successfully replaced 7 secrets in the base config -> MLAPI_SECRET_KEY, AWS_SECRET_ACCESS_KEY, ZM_PORTAL, AWS_ACCESS_KEY_ID, ZM_PASSWORD, ZM_USER, PLATEREC_ALPR_KEY
config:mlapi: substituting vars for the base config keys
config:mlapi: successfully replaced 1 default sub var in the base config -> 'base_data_path'
config:mlapi: starting {{var}} substitution
config:mlapi: successfully replaced 50 sub vars in the base config -> platerec_stats, yolo4_object_framework, alpr_key, face_train_model, yolo4_object_weights, known_images_path, tpu_object_weights_mobilenetv2, unknown_images_path, save_unknown_faces, frame_strategy, same_model_sequence_strategy, face_detection_pattern, past_det_max_diff_area, platerec_min_score, gpu_max_processes, tpu_max_processes, object_min_confidence, yolo4_object_processor, model_sequence, tpu_face_weights_mobilenetv2, smart_fs_thresh, face_recog_dist_threshold, tpu_object_labels, max_detection_size, tpu_object_framework, match_past_detections, disable_locks, cpu_max_processes, openalpr_cmdline_min_confidence, fp16_target, contained_area, alpr_detection_pattern, face_num_jitters, tpu_object_weights_mobiledet, gpu_max_lock_wait, face_upsample_times, yolo4_object_config, frame_set, yolo4_object_labels, save_unknown_faces_leeway_pixels, openalpr_cmdline_params, show_models, face_detection_framework, platerec_min_dscore, tpu_min_confidence, tpu_max_lock_wait, face_model, openalpr_cmdline_binary, cpu_max_lock_wait, object_detection_pattern
config:mlapi: attempting to build an overrode config from monitor '1' overrides
config:mlapi: monitor '1' overrides, starting {[secrets]} substitution
config:mlapi: no secrets were replace during overrode config build
config:mlapi: no vars were replace during overrode config build
config:mlapi: found a polygon for defined zone 'front_yard' -> [(0, 877), (2170, 553), (3822, 1131), (3822, 2141), (0, 2159)]
config:mlapi: detection pattern for monitor '1' defined zone 'front_yard' -> (person|dog|cat)
config:mlapi: these are they keys that were overrode in the base config -> object_detection_pattern, frame_set, model_sequence
config:mlapi: these are the keys from overrides that did not have a base value to override but are now in the config -> person_min_confidence
ephemeral detection_patterns = {'front_yard': '(person|dog|cat)'}
self.polygons = {1: {'front_yard': [{'polygon': [(0, 877), (2170, 553), (3822, 1131), (3822, 2141), (0, 2159)], 'pattern': '(person|dog|cat)'}]}}
perf:config:mlapi: 0.000297 seconds for monitor '1' overrode config build
config:mlapi: attempting to build an overrode config from monitor '3' overrides
config:mlapi: monitor '3' overrides, starting {[secrets]} substitution
config:mlapi: no secrets were replace during overrode config build
config:mlapi: no vars were replace during overrode config build
config:mlapi: found a polygon for defined zone 'parking_pad' -> [(5, 408), (660, 175), (1695, 100), (1910, 275), (1910, 1070), (0, 1080)]
config:mlapi: detection pattern for monitor '3' defined zone 'parking_pad' -> (person|dog|cat)
config:mlapi: these are they keys that were overrode in the base config -> frame_set, max_detection_size, object_detection_pattern
config:mlapi: these are the keys from overrides that did not have a base value to override but are now in the config -> person_min_confidence
ephemeral detection_patterns = {'parking_pad': '(person|dog|cat)'}
self.polygons = {3: {'parking_pad': [{'polygon': [(5, 408), (660, 175), (1695, 100), (1910, 275), (1910, 1070), (0, 1080)], 'pattern': '(person|dog|cat)'}]}}
perf:config:mlapi: 0.000216 seconds for monitor '3' overrode config build
config:mlapi: attempting to build an overrode config from monitor '4' overrides
config:mlapi: monitor '4' overrides, starting {[secrets]} substitution
config:mlapi: no secrets were replace during overrode config build
config:mlapi: no vars were replace during overrode config build
config:mlapi: found a polygon for defined zone 'back_yard' -> [(0, 547), (824, 246), (1511, 161), (1706, 1070), (0, 1080)]
config:mlapi: these are they keys that were overrode in the base config -> object_detection_pattern, frame_set
config:mlapi: these are the keys from overrides that did not have a base value to override but are now in the config -> person_min_confidence
ephemeral detection_patterns = {'back_yard': '(person)'}
self.polygons = {4: {'back_yard': [{'polygon': [(0, 547), (824, 246), (1511, 161), (1706, 1070), (0, 1080)], 'pattern': '(person)'}]}}
perf:config:mlapi: 0.000197 seconds for monitor '4' overrode config build
config:mlapi: attempting to build an overrode config from monitor '5' overrides
config:mlapi: monitor '5' overrides, starting {[secrets]} substitution
config:mlapi: no secrets were replace during overrode config build
config:mlapi: no vars were replace during overrode config build
config:mlapi: these are they keys that were overrode in the base config -> object_detection_pattern, model_sequence, frame_strategy, frame_set, face_upsample_times, face_num_jitters
config:mlapi: these are the keys from overrides that did not have a base value to override but are now in the config -> same_model_seq_strategy
ephemeral detection_patterns = {}
self.polygons = {}
perf:config:mlapi: 0.000148 seconds for monitor '5' overrode config build
perf: 0.288383 seconds -> total time to build both ZMES and MLAPI config objects with full configs for each overrode monitor
there is a copy of what the config file looked like at first read, a copy of the config with secrets and vars substituted before any monitor overrides
there is also a copy of a config for each monitor that is in the per monitor section that has been overridden. You can now override ml_sequence with {[secrets]} and {{vars}} embedded in substrings wherever you want! There are some keys that are off limits for per monitor overrides though, like base_data_path or other core configuration params that per monitors should not be touching.
there is also a hash function that saves the config and secrets file hash to check if the config or secrets file has changed on disk, if it has you can rebuild the config
Process finished with exit code 0
This will make the configuration an object that can be passed around freely and modified/updated easier. Just making this change will clean up a lot of the code base, make the code a lot easier to understand and more than likely optimize things a bit. I am starting to slowly rewrite the system piece by piece but will make improvements to the existing repos.
Edit: I have mlapi working with the new config system already, tmrw I will convert zmes over. It is disturbing how much code is removed using the new config system, I'm optimizing the best I can
