Record High-res H264 streams 24/7 with low CPU Load

[tuxmos]

Oh. My cameras actually have no audio either, but offer in the encoder settings to encode an audio track in the stream.

There is still a difference.
If I start zmrecord.sh <MonitorId> alone in the console, the system load on a core is around 3% and the script does not even appear in HTOP, only ffmeg.
If I start the script for the same monitor as service, the system load on four cores is approx. 10% and the script can be seen as in my screenshot.

Could this be that version 0.7 accesses the database much more frequently than 0.3?

[evolotion]

Hi, my mysql database has a password (as I believe they all do? on a normal ZM installation)

I cannot get the script to work even if I put a -u and --password= in every mysql call in the script in plaintext .

Yes I realise its dumb to put cleartext passwords in a script, but wanted to see if this worked.

Im not a linux noob but i am also not great with scripting and am an absolute 100% noob with mysql.

If this is an intentional barrier to entry, no hassle will keep an eye on this and see if it makes it into the main zoneminder setup, Tho if someones inclined to give me a pointer Id sure appreciate it.

cheers,
denis.

[evolotion]

ok, got it working by running by sudo -u www-data zmrecord.sh 7

and changing all the mysql lines to include login details

for example mysql -NBqr -u zmuser -pzmpass zm


anyways it records the camera and adds events to the log and the cpu usage is next to nothing, my concern before I add it as a service and set it up across my hi-res treams is this :

Code: Select all

mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
/usr/local/bin/zmrecord.sh: line 283: 0: Permission denied
mysql: [Warning] Using a password on the command line interface can be insecure.

line 283 is marked with the * .. not really sure whats happening here but when the script is running, but function changes to capturing, it shows the correct frame rate, but does not show the bandwidth.

Code: Select all

# Keep the monitor status up to date so fps etc can be viewed in the zm console
function updateMonitorStatus() {
        MONITOR_ID="$1"
        STATS_FILE="$2"
        SHUTTINGDOWN="$3"
        STATUS="NotRunning"
        FPS=0
        BW=0
        if ! [ "$SHUTTINGDOWN" = "1" ]; then
                if [ -f "$STATS_FILE" ]; then
                        IFS=' ' read -r TS FPS BW FRAME <<< `cat "$STATS_FILE"`
                        if  [ -z "${TS//[0-9]}" ] && [ -n "$TS" ]; then
                                AGE=$(($(date +%s)-TS))
                        else
                                AGE=1000
                        fi
                        if [ "$AGE" -lt 4 ]; then
                                STATUS="Connected"
                        fi
                fi
        fi
        # Calc CaptureBandwith in the same way as ZM in zm_monitor.cpp:
        #       unsigned int new_capture_bandwidth = (new_camera_bytes - last_camera_bytes)/(now-last_fps_time);
        # Unfortunately, BASH can't trap a divide by zero so we have to check
        # Get Filesize for Bandwidth calc.  Ffmpeg buffers quite a lot so see if the file has changed
        # size before calculating the Bandwidth.
        FILESIZE=`stat -c%s "$NEW_VIDEO_PATH" 2>/dev/null`
        if [ ${FILESIZE:-0} -gt ${LAST_FILESIZE:-0} ]
        then
                ELAPSED_SECONDS=$(($TS - $LAST_FPS_TIME))
*                if [ $ELAPSED_SECONDS > 0 ]
                then
                        FILESIZE_INCREASE=$(( ${FILESIZE:-0} - ${LAST_FILESIZE:-0} ))
                        LAST_BW=$(( $FILESIZE_INCREASE / $ELAPSED_SECONDS ))
                        LAST_FPS_TIME=$TS
                        LAST_FILESIZE=$FILESIZE
                fi
        fi
        SQL="   REPLACE INTO Monitor_Status (MonitorId, Status, CaptureFPS, CaptureBandwidth) 
                VALUES 
                ('$MONITOR_ID','$STATUS', '$FPS', ${LAST_BW:-0})"
        echo $SQL | mysql -NBqr -u zmuser -pzmpass zm
}

This is absolutely superb btw, thankyou!

EDIT :- ok running the script as root runs perfect and updates the bandwidth. superb

[calmor15014]

Thanks a ton for this.

I had to make a number of tweaks to get this to work on a remote MySQL installation, but was able to get it to run successfully with only recurring "Using a password on the command line interface can be insecure" warnings. It's pretty hack-ish but I'd be happy to share if you like.

I originally wanted to read the zm conf.d files but I settled for just adding some optional config variables at the head. I obviously couldn't test a local server, but the remote one works.

[ebeng]

Is there also a possibility where we could play these events in the Zoneminder webpage?

https://ibb.co/TYL1NVz

update: I can ofcourse download the file as mp4, but option to view that file would be awesome.

[ebeng]

also any idea guys why these recordings are deleted after 149 times?
this is exactly 24 hours of recording which i'm having at the end( it deletes the last 4, so 145 events are left)

my disk is only 10% full... I also dont have any other filter than the default “purge” filter, even disabling this (run in the background to disable) did not help, even with debugging, I can;t see anything in the logging...

In the zmrecord.sh script, there is no any place where an event could be deleted, or am I seeing thing over?


Update-1:
I've checked some audit logs to the oldest file which was there:

Code: Select all

ubuntu@zoneminder_v18:~$ grep 323704 /var/log/audit/audit.log
type=PATH msg=audit(1592224034.271:1651): item=0 name="/var/cache/zoneminder/events/1/2020-06-14/323704/" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1592224034.271:1652): item=0 name="/var/cache/zoneminder/events/1/2020-06-14/323704/" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1592224073.251:1687): item=0 name="/var/cache/zoneminder/events/1/2020-06-14/323704/" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1592224073.255:1688): item=0 name="/var/cache/zoneminder/events/1/2020-06-14/323704/" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1592224549.018:1725): item=0 name="/var/cache/zoneminder/events/1/2020-06-14/323704" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1592224549.018:1726): item=0 name="/var/cache/zoneminder/events/1/2020-06-14/323704" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CONFIG_CHANGE msg=audit(1592224549.018:1727): auid=4294967295 ses=4294967295 op=updated_rules path="/var/cache/zoneminder/events/1/2020-06-14/323704/323704-video.mp4" key=(null) list=4 res=1
type=PATH msg=audit(1592224549.018:1728): item=1 name="323704-video.mp4" inode=13505121 dev=fd:00 mode=0100644 ouid=33 ogid=33 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CONFIG_CHANGE msg=audit(1592224549.134:1729): auid=4294967295 ses=4294967295 op=remove_rule path="/var/cache/zoneminder/events/1/2020-06-14/323704/323704-video.mp4" key=(null) list=4 res=1
type=PATH msg=audit(1592224549.126:1730): item=1 name="/var/cache/zoneminder/events/1/2020-06-14/323704" inode=46923778 dev=fd:00 mode=040755 ouid=33 ogid=33 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

Checking out who this user is:

Code: Select all

ubuntu@zoneminder_v18:~$ id -u www-data
33

But how is this possible? Some bug in v1.34.15 ?

[russell_i_brown]

I've not really noticed before but yes... mine does the same thing - shows a max of 146 recordings for the injected captures.

At a guess it'll be something to do with the 'unofficial' way zmrecord is injecting SQL records; but that's
just a guess.

However, I don't run zmaudit on my Big RAID of 24/7 recordings; I just run the following simple
script daily from cron to maintain some headroom. My 11TB RAID stores a couple of months worth
of 3840x2160 streams for 10 cameras.

It's all very crude but has been doing the job for me for years.

Code: Select all

#!/bin/bash
#       For some reason, ZM touches updates the modtime on directories
#       so a simple find doesn't work.
#
#       Make sure this much is free on the RAID.
PERCENT=95
# Min days to retain
MIN_DAYS=16
#
# Number of days to search back before MIN_DAYS
PREVIOUS_DAYS=300
LOGFILE=~/df.log

echo "---- /mnt/bigraid ------" >> $LOGFILE
date >> $LOGFILE ; df -h >> $LOGFILE

while [ `df --output=pcent /mnt/bigraid | tr -dc 0-9` -gt $PERCENT -a $PREVIOUS_DAYS -gt 0 ]
do
        THIS_DAYS="$(($MIN_DAYS+$PREVIOUS_DAYS))"
        rm -rf /mnt/bigraid/zm/*/`date --date="$THIS_DAYS days ago" +"%Y-%m-%d"`
        PREVIOUS_DAYS="$(($PREVIOUS_DAYS-1))"
done

df -h >> $LOGFILE
echo "---- /mnt/bigraid ------" >> $LOGFILE

[ebeng]

I've not really noticed before but yes... mine does the same thing - shows a max of 146 recordings for the injected captures.

At a guess it'll be something to do with the 'unofficial' way zmrecord is injecting SQL records; but that's
just a guess.

However, I don't run zmaudit on my Big RAID of 24/7 recordings; I just run the following simple
script daily from cron to maintain some headroom. My 11TB RAID stores a couple of months worth
of 3840x2160 streams for 10 cameras.

It's all very crude but has been doing the job for me for years.

Code: Select all

#!/bin/bash
#       For some reason, ZM touches updates the modtime on directories
#       so a simple find doesn't work.
#
#       Make sure this much is free on the RAID.
PERCENT=95
# Min days to retain
MIN_DAYS=16
#
# Number of days to search back before MIN_DAYS
PREVIOUS_DAYS=300
LOGFILE=~/df.log

echo "---- /mnt/bigraid ------" >> $LOGFILE
date >> $LOGFILE ; df -h >> $LOGFILE

while [ `df --output=pcent /mnt/bigraid | tr -dc 0-9` -gt $PERCENT -a $PREVIOUS_DAYS -gt 0 ]
do
        THIS_DAYS="$(($MIN_DAYS+$PREVIOUS_DAYS))"
        rm -rf /mnt/bigraid/zm/*/`date --date="$THIS_DAYS days ago" +"%Y-%m-%d"`
        PREVIOUS_DAYS="$(($PREVIOUS_DAYS-1))"
done

df -h >> $LOGFILE
echo "---- /mnt/bigraid ------" >> $LOGFILE

I dont mind that Zoneminder is deleting the event in Zoneminder, but it also deletes the .mp4 files of the recording + the date of the folders which were created by the script

Bytheway; why do you need that script if all the events already are deleted by the Filter in Zoneminder?
and how to turnof the zmaudit? in 1.34.15?

[russell_i_brown]

ZM isn't deleting my 24/7 recordings; they're all sitting there in the filesystem which is why I need that crontab triggered script to stop it filling up.

I store the 24/7 high-res mp4's in a different storage area (#2) on a mounted filesystem and tell zmaudit to only deal with storage area #1 which is where it's storing normal event stuff..

I call zmaudit from cron like so:

Code: Select all

20 * * * * zmaudit.pl --storage_id=1

HTH

[ebeng]

Unchecking the box in the "Options-System" is not enough for ZMAUDIT to stop I see now.
After it is unchecked you have to restart Zoneminder.
I can see now that the events + the recordings are now above 180 events, for the recordings

I have to see how long it will take now, untill the first event-recording is now going to be deleted (by the FILTER in Zoneminder), hope not tooo soon and not toooo late

Thanks for the script bytheway!!
I was already using the older version, now I have upgraded to another VM, with newer ZM-version, the old script wasnt working anymore thanks for the newer one! This is now also showing directly in Zoneminder the recordings, plus the bandwidth etc!! REALLY good job!!!

[ovargaspcr]

Can I ask what OS and version are you running the scripts, I'm running on Ubuntu 16.04 and I'm getting errors when I try to run the zmrecord.sh, its funny because it puts this on the log and then it exits:

zmrecord.sh Monitor Data Error. Got: Width 1920 1080 rtsp://viewer:viewer@192.168.5.151:554/stream1 2 Event- Height ReadPath StorePath /var/cache/zoneminder/events/9

It's reading all data into Width Variable..

[trumee]

Is it possible to view the High-res streams recorded via ffmpeg in ZoneMinder interface?

[trumee]

/usr/local/bin/zmrecord.sh: line 283: 0: Permission denied

I was getting the same error as well. I replaced line 284 from

Code: Select all

                ELAPSED_SECONDS=$(($TS - $LAST_FPS_TIME))
                if [ $ELAPSED_SECONDS > 0 ]

to

Code: Select all

                ELAPSED_SECONDS=$(($TS - $LAST_FPS_TIME))
                if (( $ELAPSED_SECONDS > 0 ))

My camera has audio has well so added that -c:a aac parameter as well

Code: Select all

ffmpeg -y -loglevel error -i "$CAM_URL" -c copy -f segment -c:a aac

[trumee]

I want to copy these continuous 10 minutes mp4 files to /archive folder. I tried to setup a filter to move these files like this thread. Unfortunately, the filter is called as soon as the mp4 is opened for writing. Is there any way to make the filter wait until the event is complete?

[iconnor]

Just a heads up, in 1.35.16 (just pushed to master) we now have a DecodingEnabled setting on each monitor that turns off the decoding. So this script may or may not be necessary anymore.