Machine learning video detection

[xxrb2010]

Hi,

With all the open source machine learning frameworks around there and the progress made on video detection With those frameworks I wonder why nobody already proposed to add a new detection method based on machine learning frameworks.

Take my input as a way to start the discussion.

Amazon just released a camera that will do exactly that based on "open source". As I see Amazon as a thief or a parasite in the open source world I am surprise that anybody in this forum never propose such a new method for event detection.

Looking forward for your inputs.

Please do not discuss your view on Amazon. This is not the goal of my post. I just put mine as I took Amazon as an example and want to be transparent with my post.

[asker]

I don't think its hard at all. It most certainly isn't out of the range of skills for the ZM devs. It is essentially a matter of time. The reason I say this is machine learning based image classification has reached a point of commoditization for 'basic detection' as far as 'reasonable quality' goes. By that I mean there are a plethora of tools and frameworks, in C, in C++, in Python and in javascript that "just work" without devs having to figure out much. You basically only need to feed it new images (many of them) and it works reasonably well.

Neural nets do image classification differently from what we have been used to in Open CV. The simplest way to describe this is in OpenCV, you need to figure out the layers and steps involved in trying to break down an image into identifiable patters - that is, you figure out the filters, the transforms and the thresholds.

In neural nets, you supply the model with a whole bunch of images, mark what is "positive" (example, person) and what is not, and let the model start classifying randomly and then keep tuning its error parameters to match the original predictions. Also, interestingly, there is a concept called "transfer learning" which basically lets us use an already trained model (like inception) to learn new images and it works very well. For example, I did an experiment a year ago, where I used tensor flow to train my own new model with 1000 images of a particular classification that did not exist in inception. I made two models - one that I trained from scratch, and the other was based off inception (transfer learning). The inception model did much better.

So bottom line, doing ML image classification has pretty much wrapped itself into 'using libraries' unless you are a researcher trying to improve existing accuracy standards.

I don't think it will take more than 1 man month of time to get something like YOLO working with ZM, integrated into its core zm_monitor.cpp process. Doing it as an external trigger (OPT_TRIGGER) is even easier, and I think someone already did it.

[xxrb2010]

I think you both resume my understanding so far:

machine learning algorithms are not set and go, you have to train them by feeding them with inputs. I would argue the current zm detection method is also a try and error method to tune it correctly. At least this is what I did. But I am convince that in the long run the machine learning algorithm will beat the current manual try and error method.

Having said so, there are a lot of progress this day in the open source machine learning algorithm with on the shelf advanced libraries.

Looking forward to see if a developer might check in

[montagdude]

After having run ZoneMinder for several months, I agree that this would be a really killer feature even if the current motion detection settings are retained. Imagine that when you view an event, there are buttons to "Mark real alarm" and "Mark false alarm," which if you click, ZoneMinder will add the alarm images from that event into a machine learning database and start retraining the algorithm. Eventually, after feeding the algorithm many events, false alarms should be almost completely eliminated. Of course, I have no idea how computationally expensive it would be for the algorithm to analyze potential alarm frames on the fly.

If I were a single man with lots of free time, I might start experimenting with this myself. Getting back to reality, I don't have any experience with imaging science, neural networks, the ZoneMinder source code at any deep level, or the time to learn all those things. But it seems like it would be a fun side project for one of the developers...

[Jarod42]

I don't think that ZM really needs a machine based learning algorithm before it can handle motion detection reliably.

Right now ZM only calculates the number of pixels that differ from one picture to the other (please correct me if I'm wrong). Here at my setup this results in false alarms, when a cloud covers the sun or when the sun moves around the house and casts a shadow, or brightens certain spots. Even noise on the video (due to lack of light) causes an alarm.
Most parts of a surveillance image never change, in terms of their outline. So they remain pretty much static. So ZM needs something to distinguish the moving object form the more or less static background.
Searching the internet for "master thesis reliable motion detection" brings up a whole bunch of PDFs.
Even github seems to be a source of code from people who have done some work using OpenCV and other libs.

I'd say as soon as ZM offers reliable motion detection, it is time to think about machine based learning like recognizing faces, pets & cattle...

Just my 2 cents.

[rockedge]

you'll need a bigger machine.....most users of ZM do not have the machine power to do it real time detection with Opencv or any other framework. You could upload frames to google or amazon but it wouldn't be in real time.

[asker]

An update on this:

1. I've got both YoloV3 and tiny yolo working on my ZM machine through OpenCV reporting alarms almost real-time via the event notification server (within ~3-4 seconds or less of the alarm being triggered) . My system doesn't have a GPU but is a workstation class linux machine running ubuntu. I now run this regularly at home and get incredible results instead of rubbish being pushed to my phone. I use the standard YoloV3 model not tiny yolo. I tried both and found YoloV3 to be more reliable (obviously). Given that the detection triggers after motion is detected, the load is not too much.

Read more here for a brief and here for a detailed explanation and with code here

Performance wise, I'm seeing acceptable performance on a non GPU machine. See performance notes here

I've been digging further into lower powered devices, and I do believe you can achieve usable performance on RPI 3 too using optimized builds like this one.

I'd strongly recommend folks to try it out, get things working and then look for optimized builds/models. Practical results may surprise you.

[montagdude]

Awesome work. I will have to try it out when I get a chance. I don't even have any cameras outdoors, but it's still not possible to eliminate false alarms on cloudy, windy days, so this seems really promising.

[montagdude]

I have implemented something similar to asker in my own notification daemon. It uses the API and polls for new events at a user-specified frequency (by default, 3 seconds). When ZoneMinder is not running, it checks (by default) every 15 seconds to see if it has started up. If using object detection, it runs the maxscore frame from each event through OpenCV+YOLO v3. I haven't tried zmeventserver, but by the looks of it this daemon is a lot simpler, and I'm sure less featureful. Nevertheless, it is working well for me, and this method so far has been great for eliminating false notifications. My server is a low-powered mini PC, and detection takes about 5-7 seconds while ZoneMinder is running (about 3-4 when it's not running, FWIW). To deal with the delay, the daemon can be configured to temporarily suspend detection for awhile after a positive detection has occurred and just push notifications as fast as possible instead. The code is here, feel free to try it out:

https://github.com/montagdude/zoneminder-notifier

Edit: I just tried disabling Spectre mitigations in the kernel, and now my object detection time has dropped down to 3-4 seconds with ZoneMinder running... I think I will keep it this way.

[russellhq]

I came across this GitHub which I think might make it easy to integrate machine learning into Zoneminder.

https://github.com/robmarkcole/coral-pi-rest-server/

It runs on a Pi and detection speeds can be sped up significantly by plugging in a Coral USB Accelerator which has a processor designed just for the tasks of machine learning! I have one which I've been testing and it'll detect objects in my images in less than a second. Set up correctly it'll procees images in around 50ms

https://medium.com/@aallan/benchmarking ... 3f13942245

https://coral.withgoogle.com/products/accelerator/

The server works by posting it an image and it returns what it finds in a standard format.

[gurkburk76]

Bump on the coral usb stick, i just got one and would like to use it with zoneminder