Script to convert all events to daily videos

[mikeyintn]

One of my older ZM servers is starting to fill up and I decided it was time to rebuild it. We're required to retain all footage and I wanted to make it a little more accessible if anyone needed to look up events from back over the past four years. Here's a quick-n-dirty bash script I wrote to encode each day's events to a single video file by monitor using ffmpeg. Later, I modified it to run as a cron job to only dump out the previous week's worth of footage and I put a comment in the file how to do this if desired. I was getting about 18:1 compression ratio when comparing the original images with the encoded footage which still looked pretty sharp. But of course you can always adjust this it to be more or less compressed to your liking.

zm_export.sh.zip

(1.57 KiB) Downloaded 2434 times

Hopefully it can help someone else who needs to export all the footage to nearline storage or whatnot.
If you have ways of making it better, please do so.. and re-attach I'd like to see what you did!

[alabamatoy]

Very very cool. Thank you greatly.

[alabamatoy]

New, somewhat improved, albeit possibly slower version. This takes as input a numerical month and day and exports all the found events to 1-hour long AVI files. I have been using it for some time now, and it seems to run reliably. I have another very short script which calls it with the correct month and day, and then once its done it deletes the no-longer-necessary events, and also deletes AVI files which are older than a predetermined age, to prevent the drive from filling up. You may need to make adjustments to the ffmpeg call for your specific scenario and settings. Choosing the "verbose" setting in the script is fun to see all the stuff happening, but it produces HUGE text output from the ffmpeg compilation.

https://drive.google.com/open?id=0BytuB ... U96bjVMR3c

Please LMK if you find issues or problems with it. I hope some find this useful. Thanks to mikeyintn for the starting point from which to work.

[alabamatoy]

Due to a very recent change to ffmpeg, you need to add "-safe 0" to the ffmpeg command line in the script, otherwise it fails with an "unsafe filename" error. Apparently a recent change to ffmpeg added functionality that wont allow wildcard filenames by default, so you have to turn the safe filename functionality off.

[Falka]

One of my older ZM servers is starting to fill up and I decided it was time to rebuild it. We're required to retain all footage and I wanted to make it a little more accessible if anyone needed to look up events from back over the past four years. Here's a quick-n-dirty bash script I wrote to encode each day's events to a single video file by monitor using ffmpeg. Later, I modified it to run as a cron job to only dump out the previous week's worth of footage and I put a comment in the file how to do this if desired. I was getting about 18:1 compression ratio when comparing the original images with the encoded footage which still looked pretty sharp. But of course you can always adjust this it to be more or less compressed to your liking.

zm_export.sh.zip

Hello!
Thank you for sharing the script!

I got stuck at:

Code: Select all

----- this is the directory and filename to save to ------
/mnt/sda1/ 2016-11-06_cam_12.avi
ffmpeg version N-80901-gfebc862 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.8 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
  configuration: --extra-libs=-ldl --prefix=/opt/ffmpeg --mandir=/usr/share/man --enable-avresample --disable-debug --enable-nonfree --enable-gpl --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-decoder=amrnb --disable-decoder=amrwb --enable-libpulse --enable-libfreetype --enable-gnutls --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libvorbis --enable-libmp3lame --enable-libopus --enable-libvpx --enable-libspeex --enable-libass --enable-avisynth --enable-libsoxr --enable-libxvid --enable-libvidstab
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 47.100 /  6. 47.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[concat @ 0xa1ae5c0] Unsafe file name '/var/cache/zoneminder/events/12/16/11/06/00/27/36/%05d-capture.jpg'
/tmp/dirs_to_process: Operation not permitted

Can you help me solve this?

Regards,
Falka

Ps.:

Alabamatoy's script also fails at:

Code: Select all

Starting at 2016. nov. 22., kedd, 16:00:44 CET
looking for month = 11 and day = 21

rm: cannot remove '/tmp/*.list': No such file or directory
Finding...

Sorting...

Looking for specific files to compile...


./zm_export_1day.sh: line 148: /tmp/hours.list: No such file or directory

Even if i manually create the hours.list file:

Code: Select all

Starting at 2016. nov. 22., kedd, 16:04:28 CET
looking for month = 11 and day = 21

Finding...

Sorting...

Looking for specific files to compile...


./zm_export_1day.sh: line 148: /tmp/hours.list: No such file or directory

[Vaso]

Alabamatoy's script is really ugly as mentioned inside )))
It will not work for you unless you have the same directory structure as its author has.
1.
If you have events_dir like this:
/usr/share/zoneminder/www/events versus author's /zmdata/events
you should shift arguments number by 3 in lines 88--94

Code: Select all

    CAM="$(echo $EVENTDIRS |awk -F/ '{print $4}')"
    YEAR="$(echo $EVENTDIRS |awk -F/ '{print $5}')"
    MONTH="$(echo $EVENTDIRS |awk -F/ '{print $6}')"
    DAY="$(echo $EVENTDIRS |awk -F/ '{print $7}')"
    HOUR="$(echo $EVENTDIRS |awk -F/ '{print $8}')"
    MIN="$(echo $EVENTDIRS |awk -F/ '{print $9}')"
    DUNNO="$(echo $EVENTDIRS |awk -F/ '{print $10}')"

    to

    CAM="$(echo $EVENTDIRS |awk -F/ '{print $7}')"
    YEAR="$(echo $EVENTDIRS |awk -F/ '{print $8}')"
    MONTH="$(echo $EVENTDIRS |awk -F/ '{print $9}')"
    DAY="$(echo $EVENTDIRS |awk -F/ '{print $10}')"
    HOUR="$(echo $EVENTDIRS |awk -F/ '{print $11}')"
    MIN="$(echo $EVENTDIRS |awk -F/ '{print $12}')"
    DUNNO="$(echo $EVENTDIRS |awk -F/ '{print $13}')"
    

2.
In line 112 "/zmdata/events" have to be replaced with "$events_dir"

3.
In the same line "00" makes impossible to convert events that have "seconds" != "00".
Thats why "DUNNO" should be reincarnate to "SECONDS" and paste in place of "00".

4.
Search method is nightmare!!
Script searches ALL events. What for? We need only ONE day!!

I have resolve above issues in attached script. Try it.
Hope this helps.

P.S. I edit this script in haste so some correction may be a little half-assed. Feel free to improve.

[ConcreteRooster]

Here's yet another variation on the same theme, this time in Python.

This script works by taking in a top-level directory to search for JPG files; those files will be concatenated into a video file using ffmpeg. Additionally, if requested, each JPG can be run through the ImageMagick program "convert" to scale the size up or down. (Now that I'm writing this I suspect ffmpeg has a way to do this internally; I've been using "convert" for years so it was natural/intuitive for me to use that.) You can disable this by setting the scale argument to 0 or 100 (default is 50).

Here's the --help for an overview:

Code: Select all

$ ./zm_mkvid.py -h
usage: zm_mkvid.py [-h] [--event-dir EVENTDIR] [--verbosity LEVEL] [--dry-run]
                   [--scale SCALE_PCT] [--prefix PREFIX]

Concatenate Zoneminder event image files into video

optional arguments:
  -h, --help            show this help message and exit
  --event-dir EVENTDIR, -d EVENTDIR
                        Top level zoneminder event directory to walk for image
                        files
  --verbosity LEVEL, -v LEVEL
                        Set verbosity/logging level. Valid options: ['INFO',
                        'DEBUG', 'CRITICAL', 'WARNING', 'ERROR']
  --dry-run, -n         Don't actually do anything, just pretend.
  --scale SCALE_PCT, -s SCALE_PCT
                        Specify scale factor for ImageMagick convert.
  --prefix PREFIX, -p PREFIX
                        Prefix for output video file name

So, example command (all on one line):

Code: Select all

./zm_mkvid.py --event-dir=/var/lib/zoneminder/events/2/17/02/02/ --verbosity=DEBUG --scale 75 --prefix=capture_monitor2

This would create a video file named "capture_monitor2_20170202_HHMMSS-20170202_HHMMSS.mkv", where the two HHMMSS fields are replaced by the times of the first and last events for Feb 2, 2017. Since verbosity is set to DEBUG, it will print a bunch of stuff that most people won't be interested in. The images will also be scaled to 75% of their original size before being fed into ffmpeg.

Note I had to append ".txt" to the attachment filename for the forum software to accept it. Obviously you'll want to remove the .txt and "chmod +x".

This was developed/tested under CentOS 7.3, using ZM 1.30.0 from zmrepo with the ImageMagick package installed (provides the "convert" program). It "should" work on other platforms, but might require some tweaking.

Hope someone finds it useful!

zm_mkvid.py.txt

Remove .txt extension

(9.88 KiB) Downloaded 1559 times

[ConcreteRooster]

Here's a slightly updated version of the script above. Tweaks:

• --event-dir argument can be passed multiple times
• --event-dir argument(s) is(are) interpreted as shell glob pattern(s)
• --help output has been updated to indicate defaults (e.g. 50% scaling)

Enjoy!

[ConcreteRooster]

Another update:

• Bugfix: the previous --event-dir enhancements could create a situation where the scaled files (or symlinks if not scaling) had colliding filenames. Fixed.
• Enhancement: added a --no-cleanup option which, if used, will stop the script from cleaning up temp/intermediate files (likely only useful for debugging)

[ConcreteRooster]

Here's another update, one minor enhancement and one major:

• Added --framerate CLI option. I didn't test this too much, but it's basically just a pass-through to the ffmpeg -framerate option. The default is 15. Play with this if the videos created with this tool don't play back at the speed you expect.
• Stripped out all the ImageMagik/convert stuff in favor of doing any requested scaling via ffmpeg. In my quick tests, this was both faster and resulted in a smaller video file. It's also one less dependency and makes the script simpler. So no downside as far as I can tell, other than it hasn't been tested as much.

Enjoy, please let me know if you find any bugs or issues!

[ConcreteRooster]

Small update, you might call it a bugfix: I had a sanity check to avoid duplicate JPG files. But if you're creating a video from a huge number of JPG files (say, a whole day's worth when using Record Mode), the performance degraded to unacceptable levels. (My process was still running after 24 hours when I killed it, just trying to build the file list.)

Anyway, I commented-out the check, and now things run much faster, particularly when making videos from many images.

I can't think of any more changes/enhancements I want to make to this, so hopefully this is a semi-final version. (Famous last words, right? )

Enjoy!

[johnnytaco]

This is awesome! I just tested by creating a video of the entire day for my driveway camera. Worked perfectly. Thanks for sharing.

jt

[ConcreteRooster]

This is awesome! I just tested by creating a video of the entire day for my driveway camera. Worked perfectly. Thanks for sharing.

My pleasure, glad you enjoy it!

One more small update. This is an enhancement/new feature only, so no need to download unless you want the feature. I added a "--yesterday" argument, which works similar to the --event-dir argument. But instead of passing the full event directory, you just pass the event directory up to the monitor name only. The remaining portion of the event directory is the date, which is automatically computed as yesterday's date. This is really just a convenience function.

With this feature, I created a wrapper script that makes a full-day archive of yesterday's footage. I put the wrapper script in cron. In particular, I am making these archives from my always-recording "substreams" that my cameras provide. (The substreams are secondary, lower-resolution streams.)

My wrapper script looks like this, I named it mkarchive.sh:

Code: Select all

#!/bin/bash

ZM_MKVID=/path/to/zm_mkvid.py
ZM_MONITORS='Front_Porch_Substream Rear_Porch_Substream'
ZM_EVENTS_DIR=/var/lib/zoneminder/events
NOW_STRING=$( date +%Y%m%d-%H%M%S )
DEST_DIR=/path/to/video_archive

cd ${DEST_DIR}

for monitor in ${ZM_MONITORS} ; do
    log=/tmp/zm_mkvid.${monitor}.${NOW_STRING}.txt
    ${ZM_MKVID} \
        --scale=0 \
        --yesterday=${ZM_EVENTS_DIR}/${monitor} \
        --prefix=${monitor} > ${log} 2>&1
done

And then my simple cron entry:

Code: Select all

00 03 * * * /path/to/mkarchive.sh

I have two monitors. Every day at 3:00am I create a archive videos of yesterday's footage in their entirety.

Finally, within Zoneminder, I have a filter configured for substream events older than a certain time period. With that filter, I can delete the old events since everything has been archived with video. I could make the filter automatically delete, but right now I run it manually once a week or so. That's really the only thing I feel I'm missing at this point, some extra error checking to make sure everything works as expected.

I've been running with this setup for a few weeks now, so this time I really don't plan on any more changes to the script.

[jaysix79]

hello ConcreteRooster

is there a way to change the date format ? i tried changing them with success

date_format='%b%d%Y-%H:%M:%S'

i wanted to write the video in this format

XXXX.from.May.20.2017-Time.10.27.00-to. May.21.2017-Time.10.27.00

[ConcreteRooster]

hello ConcreteRooster

is there a way to change the date format ? i tried changing them with success

date_format='%b%d%Y-%H:%M:%S'

i wanted to write the video in this format

XXXX.from.May.20.2017-Time.10.27.00-to. May.21.2017-Time.10.27.00

Hi, sorry I didn't respond sooner - I haven't been checking this thread lately.

At any rate, this is a fairly easy change. I am making the output file naming completely user-configurable. I already have the changes made, am testing now. Assuming no issues, I'll post the updated version shortly.

(Note: that date_format you changed is for the Python logger module, and completely independent of the output filename.)