Hi,
I have dabbled with ZoneMinder in the distant past, but at the time I had difficulty installing it. For now, I'm using the default TrendNet surveillance software, but am planning to make the jump to ZoneMinder. I am a programmer and very interested in AI. I am sure many of the ideas I'm going to mention have been thought of or planned to integrate. I am a systems integrator by nature, and if I could find someone to work with, I would be interested in trying to integrate these capabilities.
My question is, can anyone integrate / help me integrate the following tools into ZoneMinder? First, this integration of YOLO is awesome! I hope it is the latest version of YOLO. Object detection is great - I don't know what kind of inference mechanism for sensor integration ZoneMinder uses, but being able to tell if a person is in view of the camera and triggering a callback is a great idea. For instance, if you lived in a neighborhood that had a high crime rate and you were concerned about burglaries - this could transform ZoneMinder from a surveillance system into an intrusion prevention system where motion detection alone fails, for instance by disambiguating people from pets. Another use of object detection is inventory management, for instance, if you brought in a particular item, the system could mark the route the item took (which cameras, or even spatial information). Another application would be monitoring the health of pets, to detect anomalies in their daily routines.
Anomaly detection also have obvious security applications. For instance, if someone is casing your house, or you have a particular neighbor you're trying to avoid, or such. For such, I've heard that gait recognition is invaluable. There is a gate recognition system that I imagine may be able to work here:
https://github.com/marian-margeta/gait-recognition
This seems to be the best option available on GitHub, from a cursory glance. I briefly tried setting it up but I am not a Python programmer and still have to figure out how to get the frames variable set. I am not sure how this would be integrated, but when I found this thread here about ZoneMinder YOLO integration, I became more optimistic seeing as you have managed to get them to work together. I do not know what kind of inborn intelligence comes with this Gait Recognition repository. I wonder if it can handle previously unseen gaits, and create new identities based on newly observed gaits, which could be ordered by frequency of occurrence and a GUI given to the user to add any other information such as names or such.
While gait recognition in itself is useful in providing identification, what would really help is integration of anomaly detection with gait recognition. This way, for instance, people walking to and from work or school would quickly be normalized, whereas the same individual appearing three times throughout the day at irregular hours would be anomalous and trigger a warning. It can also help with recording for the user their own movements both in and outside of the house. By recording these times, it can help the user to avoid predictable movements, unpredictability being a key factor in security, such as leaving for the store at the same time each week, etc, which could be exploited, such as by an ambush or a break in. Perhaps it could trigger an audible warning if the user went for the door at a predictable time. Things like that. There are many anomaly detection repositories on GitHub and many books on how to use them. And I know someone who works in that area who I could ask for help if I got stuck. Some questions are what kind of accuracy do you get at long distances and/or lower resolutions? Do the machine learning models have to be trained on the same individuals the classifiers run against?
I will just mention briefly that OpenFace face recognition could be integrated, and again I do not know how to get it to create new identities for previously unseen faces. It sets up rather quickly, but it is unclear to me after a quick glance how exactly to integrate it. This would be useful for cameras that get close ups, such as a front door camera or a camera guarding a hallway or such. Again, identification information could be used to help randomize the users movements and also for multi-factor authentication such as of a doorway. Also, it could contribute to activity recognition and also goal recognition. For instance in a smart home environment, a goal recognition system could be used to determine the purpose for a person going somewhere or doing something, and then, having figured out the most likely motives, a smart home could think about whether these are good ideas and interrupt, or suggest something like don't forget to bring the package when you drive off, etc. Here is such a system:
https://github.com/aindilis/optimal-plan-recognition
Lastly, activity detection is a good feature, if for no other reason than to add to the anomaly detection inputs. There seem to be at least a few good activity recognition systems on GitHub. Activity recognition is also important for monitoring sick or elderly patients. Such features in ZoneMinder may extend and enhance it for such other purposes.
I am not sure as to what extent existing smart home systems perform these kinds of analyses. For instance, Home-Assistant. I have been working on such a system, which can be seen here:
https://frdcsa.org/~andrewdo
Some unanswered questions remain - can a video card run multiple models at the same time, how much power does it take and is it problematic to run a video card 24x7?
Anyway, I know this is a lot to suggest, however, by adding these features, it could increase the marketability of ZoneMinder for a few reasons. Smart home technology is rapidly accelerating into the mainstream, and thus provides ample surface area for integration. Having ZoneMinder more compatible with smart home technology through the integration of computer vision capabilities could help to future proof it and market it. The availability of these computer vision tools and other AI tools such as plan and anomaly recognition now is something that simply didn't exist even 5 years ago, so there is really an explosion of options here. On the other hand, it may prove more prudent to wait it out in order for some particular combination of technologies to win out before investing effort in preliminary integrations. Whether these capabilities will be extended or whether they are one off research attempts affects the value of integrating them. Perhaps you already have APIs defined which would allow CV modules to be integrated quicker.
If the choice to integrate one or more of these technologies were made, probably best would be to do an SMEA (Success Modes Effects Analysis) to see which techniques provide the greatest benefit to the most use cases, and integrate those first.
As for what part I play, I am proficient in Perl and moderately skilled with Prolog. I have written a life planning system which combines these. In particular, I am versed in systems integration and stove piping various software systems together. I am also skilled in making operating systems packages. So I could package these tools for Debian and use Alien to help bootstrap other package formats. I think that adding these capabilities would be a force multiplier for the project.
I have many projects I am working on, but if someone were to work with me on this it would increase my motivation and I might be able to get further along - particularly someone already familiar with ZoneMinder and video programming. I would like to point out however that I have a tendency to fail to complete projects on time if at all.
Best,
Andrew