Nugget's Archive Old Events Script

If you've made a patch to quick fix a bug or to add a new feature not yet in the main tree then post it here so others can try it out.
Post Reply
User avatar
nugget
Posts: 42
Joined: Mon Dec 08, 2003 7:52 pm
Location: Sydney, Australia
Contact:

Nugget's Archive Old Events Script

Post by nugget »

Here is a script I use to archive and remove old events from the database/filesystem. The purpose of this script is to clean out old events which you're not sure you might need or not, and store them on "offline media". The script works as follows:

* is run from cron once a week
* searches through the database for all events older than ${age_in_days} (defined within the script)
* moves the contents of the selected event to an archive directory (defined as ${archive_dir} in the script)
* removes the corresponding record from the event table.

A simple report of all events removed from the database is also stored in the archive directory.

The intention is to then either move the archive directory to offline media (CD/DVD/Tape) or remove the directory after a sufficient time period.

Programmer's notes:
1. This is the first release of this script. It works quite well for my purposes. I will not be insulted if you check my work, make additions or alterations, or offer suggestions. I have been known to make mistakes...
2. There are a couple of variables defined early in the script:
age_in_days - defines the time in days you think is reasonable to remove events. Any event older than this time period is a candidate for archival. Default value is 14 days
archive_dir - this is the root directory you wish to move your archived events into. Current value is /usr/local/archive/zm/`date +%b%Y`. Events will be moved under this directory structure, into a directory archived_event/${zone_name}/${event_id}
db_name - the name of your database (probably "zm")
3. Currently I don't use any database verification to log into the db - the script runs from root's crontab, root does not have a database password on my host
4. I assume your events are stored in /var/www/html/zm/events. If not you will need to alter this path in the script.

Installation:
1. Cut and paste the code below into a script /usr/local/bin/zm_archive_old_events. chmod u+x the script
2. Make any alterations per the above notes
3. Add a cron entry for the script. I suggest once or twice a week is sufficient

Code: Select all

#!/bin/bash
# zm_archive_old_events
# Script to clean up the database/events directory on a regular basis by
# removing "old events" - an event older than ${age_in_days}
# event data is copied to an archive directory for safe keeping/later
# archival/deletion
#
#
# Author: Nugget, aka Stuart Carmichael  20/06/2004
#
 
# Define some vars used later:
age_in_days=14  # keep events online for 2 weeks, then archive
archive_dir="/usr/local/archive/zm/`date +%b%Y`"
db_name="zm"
 
archive_date="`date --date=\"${age_in_days} days ago\" +%Y-%m-%d`"
# Now get a list of all "old" events
 
mysql -D "${db_name}" << EOF > /tmp/old_event.$$
  select e.Id,m.Name,e.StartTime,e.EndTime
  from Events e
  left join Monitors m on e.MonitorId = m.Id
  where e.StartTime <="${archive_date}"
  order by m.Name,e.Id;
EOF
 
if [ `cat /tmp/old_event.$$|wc -l` -eq 0 ]; then
  echo "No matching records found, exiting..."
  exit 0
fi
 
# Using the temporary file created above, move the video files to
# the archive directory and remove the event record from the database.
# The archive directory chosen will be for the month/year for which the
# event occurred.
# An alternate method if you don't want to archive the video files is
# to simply remove the record from the Event table, and allow the
# cleanup daemon to remove the video files (requires the option
# ZM_OPT_FAST_DELETE to be turned on).
 
 
cat /tmp/old_event.$$ | grep -v "^Id" | while read event; do
  id="`echo $event|awk '{ print $1 }'`"
  zone="`echo $event|awk '{ print $2 }'`"
  event_date="`echo $event|awk '{ print $3 }'`"
  outdir="/usr/local/archive/zm/`date --date=${event_date} +%b%Y`/archived_event/$zone"
 
  if [ ! -d "$outdir" ]; then
    mkdir --parents $outdir
  fi
  
  # find the event directory, and move it to the archive location
 
  mv /var/www/html/zm/events/$zone/$id ${outdir}/
 
  # remove the record from the database
mysql -D "${db_name}" << EOF
  delete from Events where Id=${id};
EOF
 
done
 
# now copy the event listing file to the archive directory for safe keeping
outfile="$archive_dir/archived_event/archived_events.`date +%d%m%y_%H%M`"
mv /tmp/old_event.$$ $outfile
 
num_events="`cat $outfile|wc -l`"-1
echo "${num_events} archived events found and processed."
echo "Output report can be found in the following location:"
echo "$outfile"
 
# at this point the db and the events directory are cleaned up. You can either
# burn these files to CD for safe keeping, or remove them later.
szgaljic
Posts: 1
Joined: Wed Dec 10, 2008 4:19 am

Great Script.

Post by szgaljic »

Four years later I will fix a bug in this. The outdir variable here should be changed.

Code: Select all

cat /tmp/old_event.$$ | grep -v "^Id" | while read event; do
  id="`echo $event|awk '{ print $1 }'`"
  zone="`echo $event|awk '{ print $2 }'`"
  event_date="`echo $event|awk '{ print $3 }'`"
  outdir="/usr/local/archive/zm/`date --date=${event_date} +%b%Y`/archived_event/$zone" 
From

Code: Select all

outdir="/usr/local/archive/zm/`date --date=${event_date} 
To

Code: Select all

outdir="$(archive_dir)/`date --date=${event_date}
ark
Posts: 6
Joined: Wed Jul 25, 2007 8:56 pm

Post by ark »

It's worth noting that in the fix above you mean ${arhive_dir} not $(archive_dir)

I found I needed to make two more changes

I have my database protected with a password so I have a defuslts-file with access information in it

so the mysql command changed to this

FROM:
mysql -D "${db_name}"
TO:
mysql --defaults-file=~/lib/mysql/.my.cnf.zm -D "${db_name}"

i think the -D is not needed but it works
~/lib/mysql/.my.cnf.zm looks like this:
[client]
host=localhost
database=zm
user=zmuser
password=

I also had to move my archiche location (I should have made this a variable) around line 55

echo $zone/$id
mv /var/cache/zoneminder/events/$zone/$id ${outdir}/

andthen when it was running I needed to make quite a few symlinks since it seems everything just changed to 1/2/3/4 rather than the name of the input. but it appeared to work great! thank so much for a useful script!
L0key
Posts: 4
Joined: Sun Nov 29, 2009 3:03 am
Location: The Gathering Place

Post by L0key »

Hey,

I think I am on the verge of figuring this one out but I need a little help. :?

I have applied the fix szgaljic suggested, and also made the sym links that ark used.
I also have no password (for now) on the "zm" database.

However, every time I run the script I get "No matching records found.. exiting" from line 29

I ran the part of the script that queries mysql (lines 20-26 without the variables) from the cl and it populated the same target file with over 2000 entries.

Checked for typos and even tried using explicit arguments instead of variables. :(

Can anyone help
"Knowledge, without action, is useless. It's not what you learn, it's what you live."

Dr Kyla Dillard
User avatar
nugget
Posts: 42
Joined: Mon Dec 08, 2003 7:52 pm
Location: Sydney, Australia
Contact:

I'm back!

Post by nugget »

After not having a Zoneminder install for the past 4 years (moved house, renovations take precedence) I am back.

Once I complete my install I will have a look at the script & see what I can do.
kmichal456
Posts: 1
Joined: Wed Mar 10, 2010 1:16 am

Post by kmichal456 »

I'd be very interested in knowing whether this script works with current version.


Mike
jclint
Posts: 1
Joined: Wed Jun 30, 2010 10:04 pm

I think it is working for me

Post by jclint »

I think I have it working, still waiting for the script to finish archiving. However I can see the disk space going down so I am hopefull. I am using version 1.24.2.

Just a side note, it would be a nice touch to add a option in the script that lets you delete the file after so many days. With the option of not deleting it, this way you could run one script to get 2 things done.

thanks I will resubmit my success when it happens.


EDIT: OK so it worked perfectly, it put them in to a folder in /usr/local/archive/ I didn't even have to make the folder it made it. It put it as Jun2010 I am guessing for the date of the archive and then again Jun2010 for the images. That part is a bit confusing, there probably should be some identifier to diffrientiate the 2 like /usr/local/archive/archivedateJun2010/ImageDateJun2010 or /usr/local/archive/AD-Jun2010/ID-Jun2010

Anyway to clear up space i just delete one of the folders, worked like a charm. Now time to look into using this script to pull video's from the archive :) or to unarchive back to ZM.

By the way I think all of this answers the question above, Yes it does work with current release!
Jason Clint
IT Linux Admin
Post Reply