Script to convert all events to daily videos

[ConcreteRooster]

Give this a try, it should do what you want. The "--prefix" CLI arg has been removed, and replaced by the more general "--outfmt" argument. There is also a new "--timefmt" arg.

With the old --prefix arg, you set only the first part of the filename, and the rest was automatically generated for you. Now you can explicitly set the output filename format, including any leading directories and filename extension. The --outfmt argument looks for two special tokens, {START_TIME} and {END_TIME}. In the actual filename, those tokens will be replaced by the start and end times of the video that is created. You can change the format of how these timestamps are created with the --timefmt option. What you can do with --timefmt is defined by Python's time.strftime function. (This should be familiar to anyone who's done any POSIX programming or has used the shell's date command.)

So to get the format that was requested

XXXX.from.May.20.2017-Time.10.27.00-to. May.21.2017-Time.10.27.00

You'd do something like this:

Code: Select all

./zm_mkvid.py --event-dir=/var/lib/zoneminder/events/Front_Porch_Camera/17/07/12/ --timefmt="%b.%d.%Y-Time.%H.%M.%S" --outfmt='XXXX.from.{START_TIME}-to.{END_TIME}.mkv'

The defaults should be the same, so if you've never previously used the --prefix arg, then this updated version should behave no differently. If you simply want to retain the old-style "--prefix" behavior, then just append "_{START_TIME}_{END_TIME}.mkv" to the string you use for --prefix. For example:

Code: Select all

--prefix="My_Camera"

becomes

Code: Select all

--outfmt="My_Camera_{START_TIME}_{END_TIME}.mkv"

.

Other than the creation of the output filename and the CLI args, the core functionality has not changed. So if you don't need these new generalized output filename config options, there is no need to upgrade.

[jaysix79]

Code: Select all

#!/bin/bash
## to run the script

## location of the converter file
ZM_MKVID=/DIRECTORY/zm_converter.py
ZM_MONITORS_1='Dog_cam-1'
## location of the zoneminder event files
ZM_EVENTS_DIR=/var/lib/zoneminder/events
## using date as a name of log files
## sample "NOW_STRING=$( date +"DATE %b %d, %Y TIME %H:%M%P" )"
NOW_STRING=$( date +%b.%d,%Y-Time%H.%M.%S )
## Destination directory
## sample DEST_DIR=/path/to/video_archive
DEST_DIR=/OUTPUT/zoneminder/${folder}/
# making destination directory
#mkdir /DISTANATION/zoneminder/
mkdir -p ${DEST_DIR}
## Location of log files for debuging
LOG_DIR=/LOGFOLDER/zoneminder/log
# making log directory and cleaning up
sudo rm -r ${LOG_DIR}
mkdir -p ${LOG_DIR}

cd ${DEST_DIR}


for monitor in ${ZM_MONITORS_1} ; do
    log=${LOG_DIR}/zm_mkvid.${monitor}.${NOW_STRING}.txt
    ${ZM_MKVID} \
        --scale=0 \
        --yesterday=${ZM_EVENTS_DIR}/${monitor} \
	--timefmt="%b.%d.%Y-Time.%H.%M.%S" \
        --outfmt='Dog_cam-1.from.{START_TIME}-to.{END_TIME}.mkv' > ${log} 2>&1
done


exit 1

thank you it works well with this... took me a while to figure how to edit my SH file but it works. its kinda messy

another question or request would be is it possible to run this script to convert video that was taken in the last X amount of hours...... lets say i just want to convert the last 3 hours so that i can upload on my cloud and share it....

i tried changing

Code: Select all

    if args.yesterday:
        if not os.path.exists(args.yesterday):
            print >>sys.stderr, 'ERROR: {0}: does not exist, abort'.format(args.yesterday)
            sys.exit(-1)
        yesterday = time.localtime(time.time()-3600)  (IM TESTING IT TO RUN 1 HOUR)
        vid_dir  = args.yesterday + os.sep
        vid_dir += time.strftime("%y", yesterday) + os.sep
        vid_dir += time.strftime("%m", yesterday) + os.sep
        vid_dir += time.strftime("%d", yesterday)

it does convert 1 hour span or recording, but it always start at 4 am in the morning. no mater what time i change it to it always start 4 am and the only thing that change is length of recording. what i want is an HOUR from the time i run the script. thanks again for the help and sorry i dont really know a lot about script. just learning from trial and error

[ConcreteRooster]

I see one potential problem in your shell script:

Code: Select all

DEST_DIR=/OUTPUT/zoneminder/${folder}/

I don't see where you define ${folder}. That will probably evaluate to an empty string, so your files get written to /OUTPUT/zoneminder/ whereas you presumably what /OUTPUT/zoneminder/something else.

Also, just to be pedantic:

Code: Select all

for monitor in ${ZM_MONITORS_1} ; do
    log=${LOG_DIR}/zm_mkvid.${monitor}.${NOW_STRING}.txt
    ${ZM_MKVID} \
        --scale=0 \
        --yesterday=${ZM_EVENTS_DIR}/${monitor} \
        --timefmt="%b.%d.%Y-Time.%H.%M.%S" \
        --outfmt='Dog_cam-1.from.{START_TIME}-to.{END_TIME}.mkv' > ${log} 2>&1
done

I would change that to:

Code: Select all

for monitor in ${ZM_MONITORS_1} ; do
    log=${LOG_DIR}/zm_mkvid.${monitor}.${NOW_STRING}.txt
    ${ZM_MKVID} \
        --scale=0 \
        --yesterday=${ZM_EVENTS_DIR}/${monitor} \
        --timefmt="%b.%d.%Y-Time.%H.%M.%S" \
        --outfmt="${monitor}.from.{START_TIME}-to.{END_TIME}.mkv" > ${log} 2>&1
done

Notice I changed --outfmt in two ways: (1) I used ${monitor} instead of Dog_cam-1. What you have is fine, since there is only one monitor. But if you ever add another monitor, then you could possibly overwrite previously-written videos. (2) I used double quotes (") instead of single quotes ('). This allows the shell (bash) to expand the ${monitor} variable. (Not to be confused with the {START_TIME} and {END_TIME} tokens - those are evaluated by the Python script, not by the shell. In hindsight, perhaps using curly brackets wasn't the best choice.)

thank you it works well with this... took me a while to figure how to edit my SH file but it works. its kinda messy

another question or request would be is it possible to run this script to convert video that was taken in the last X amount of hours...... lets say i just want to convert the last 3 hours so that i can upload on my cloud and share it....

I think what you're asking for is a parameter similar to --yesterday, say --last-x-time=3h, or more generally, --start-datetime and --end-datetime parameters. This isn't terribly hard to do. However, my thought was that the directory structure Zoneminder uses to store images is in my mind fairly intuitive. Here's an example from my own system:

/var/lib/zoneminder/events/Front_Porch_Camera/17/07/20/07/28/54

The /var/lib/zoneminder/events might differ from one platform to the next. But below that, I believe the tree is always the same. And the format is events/monitor_id/year/month/day/hour/minute/second/<JPG files>.

Just look at the timestamps on my jpg files:

Code: Select all

$ ls -lah /var/lib/zoneminder/events/Front_Porch_Camera/17/07/20/07/28/54/
... (snip) ...
-rw-r--r-- 1 apache apache 122K Jul 20 07:28 00057-capture.jpg
-rw-r--r-- 1 apache apache 120K Jul 20 07:28 00058-capture.jpg
-rw-r--r-- 1 apache apache 121K Jul 20 07:28 00059-capture.jpg
-rw-r--r-- 1 apache apache 121K Jul 20 07:28 00060-capture.jpg
-rw-r--r-- 1 apache apache 122K Jul 20 07:28 00061-capture.jpg
... (snip) ...

See how they are all Jul 20 (implicitly 2017) 7:28, and they live in 17/07/20/07/28 tree?

That is why I specified the --event-dir parameter as a directory tree and/or glob pattern. That's why you can specify, for example, --event-dir=/var/lib/zoneminder/events/Front_Porch_Camera/17/07/20 and it will grab everything for July 20, 2017. But if I instead said --event-dir=/var/lib/zoneminder/events/Front_Porch_Camera/17/07/20/07, it would only grab everything from 7:00am to 7:59am on July 20, 2017. Don't forget you can specify --event-dir multiple times and/or use shell wildcards. (I've been a Linux guy for 20+ years, so doing shell stuff is second nature to me. It's easy for me to forget glob patterns and such may not be second nature to everyone else!)

Take a look at "man date" (or web search example usage of the date shell command). With clever use of using the output of the date command within the --event-dir argument, you can create this behavior yourself. Depending on how fancy you want to get, it could get ugly to do it in your shell wrapper script; so that's an argument for using Python's much more convenient date and time manipulation routines to do it...

The code itself isn't hard, I just have to give some thought how the user interface will look. For example, if I were to add --start-datetime --end-datetime parameters, do I then change the behavior of --event-dir? If keep --event-dir's behavior as-is, and someone uses both --event-dir and the new --start/end params, which takes precedence?

I'll give it some thought, though I'm also open to input.

[GnuSquid]

Thanks much for creating this script! I'm using it to great effect on my ZM install. One bit of feedback, and hopefully something that is fixable, the framerate option doesn't seem to do anything. Regardless of what framerate I input, the output file always defaults to the ffmpeg default at 25 fps, it's not even using the script default of 15 fps. Any ideas?

[Baylink]

This makes one video per monitor, does it? Does it process all monitors by default?

[chrisc]

Thanks much for creating this script! I'm using it to great effect on my ZM install. One bit of feedback, and hopefully something that is fixable, the framerate option doesn't seem to do anything. Regardless of what framerate I input, the output file always defaults to the ffmpeg default at 25 fps, it's not even using the script default of 15 fps. Any ideas?

I had the same problem. I have been using ffmeg version 3.3.4-2 on Ubuntu 17.10 and noticed that the -framerate does not exist it is actually -r for both input and output (ref: https://ffmpeg.org/ffmpeg.html). Setting the framerate to be the same for both input and output to match what your jpgs are stored at works perfectly. (e.g. for my 6fps images I have been doing tests on the side and found this to work great:
ffmpeg -r 6 -safe 0 -f concat -i mkvid_filelist.txt -r 6 -preset veryfast testout.mkv
I left off the -pix_fmt yuv420p as it slows things down and didnt seem to be needed for VLC or SMPLAYER (but it does make the file about 10% smaller if that interests anyone)
My command in the py code looks like this with the -r on the input and output and commented out the -pix_fmt:
---------------------------------------
cmd = [ FFMPEG,
'-r', str(self.framerate), #input and output framerate should be the same
'-safe', '0',
'-f', 'concat',
'-i', jpeg_list,
'-vf', scalestr,
'-r', str(self.framerate),
'-preset', 'veryfast',
'-y', #overwrite
# '-pix_fmt', 'yuv420p', #makes 10% smaller but slows down encode
outfilename ]
---------------------------------------
Lastly for Ubuntu I couldn't get the controlling bash script to work unless I added the word "python" to the command.
And the events are stored in /var/cache (which differs from the info provided previously for CentOS.
viz:
---------------------------------------
#!/bin/bash

ZM_MKVID=/home/chris/Downloads/zm_mkvid1.py
ZM_MONITORS='Conch Bullet_Substream' #Place all your monitors you want here and use underscore instead of spaces for names
ZM_EVENTS_DIR=/var/cache/zoneminder/events #In /var/cache for ubuntu
NOW_STRING=$( date +%Y%m%d-%H%M%S )
DEST_DIR=/home/chris/Downloads/zm_archive

cd ${DEST_DIR}

for monitor in ${ZM_MONITORS} ; do
log=/tmp/zm_mkvid.${monitor}.${NOW_STRING}.txt
python ${ZM_MKVID} --yesterday=${ZM_EVENTS_DIR}/${monitor} --scale=0 --framerate=6 --prefix=${monitor} > ${log} 2>&1
done
---------------------------------------
Hope that helps.
Cheers
Chris

[chrisc]

This makes one video per monitor, does it? Does it process all monitors by default?

See my comment about the bash script that was written previously:

ZM_MONITORS='Conch Bullet_Substream' #Place all your monitors you want here and use underscore instead of spaces for names

[solo60]

Awesome script! thanks, much appreciated.

cheers
solo.

[raftop]

this could be a one liner. Merges all videos from monitor 1 of a specific date (given as yy mm dd at the command line) into a single video.
Example how to call for merging all videos from 15th of November 2019: ./videomerge.sh 2019 11 15
change the WHATEVERS to match your case.
then save the six lines below and name the file videomerge.sh
then install melt (sudo apt install melt)
then call the script as per example above (after chmod the permissions)


events_dir="WHATEVERIS_WITHOUT_ENDING_SLASH"
archive_dir="WHATEVERIS_WITHOUT_ENDING_SLASH"
find $events_dir/1/$1-$2-$3/* -name *-video.mp4 | sort > /tmp/sortedvideos.txt
cat /tmp/sortedvideos.txt | while read f; do cp $f $archive_dir; done
cd $archive_dir
melt ./*-video.mp4 -consumer avformat:videomerger.mkv acodec=libmp3lame vcodec=libx264

[alexbba]

Hi,
A crucial change has been introduced that requires adding the "-funsafe 0" parameter to the ffmpeg command line in your script. Without this adjustment, script execution consistently fails with an error indicating "unsafe file name." This new feature appears to be intended to increase security by restricting the use of generic filenames by default, thus forcing users to explicitly disable this feature to return ffmpeg to its previous functionality, especially when handling video files