Autoclean Script for events (V 2.0)

[victor_diago]

Hi all the code is the following (cut it and paste into a text file named autoclean2.sh and then chmod +x in it) you should run it in a cron way every 30 minutes.

Code: Select all

#!/bin/bash
# This is the Second version of autoclean.sh for ZM Events.
# autoclean.sh will remove (from filesystem AND database)
# the oldest day, when your free disk amount is less than
# the value defined in "lessdiskamount"
#
# See that This WILL NOT remove ARCHIVED events
# so this is a very safe way to remove old events
#
# ZM (ZoneMinder) is the top Surveillance System in Linux
# and its manteined by Philip Coombes.
#
# Fell Free to contact me (Victor Diago) in vdiago@spymac.com
#
# With a little help we could make ZM Always Better.
#
# Victor Diago


# Put your partition (like hda1) where your events are. this is used to determine
# available space.
partition="your partition"

# Put here your DataBase Name
database="your db name (default zm)"

# Script will clean the oldest day when free space is less than XXX MB

#lessdiskamount="less disk amount to allow (IN MB)"
lessdiskamount="500"

# Your ROOT DB username
rootdb="root s db username"

# Your ROOT DB password
rootpass="root s db password"

# Now we will calculate how much free space you have actually

freeamount=`df -m|grep "$partition"|cut -c 42-50|sed s/" "/""/g`

# Now we need to calculate the last day recorded without Archive flag

lastday=`mysql -B -u "$rootdb" --password="$rootpass" -e "use zm;select StartTime from Events where Archived <> 1 limit 1;"|tail -1|cut -f1 -d " "`

while [ "$freeamount" -lt "$lessdiskamount" ];
	do
		mysql -B -u "$rootdb" --password="$rootpass" -e "use zm;delete from Events where StartTime LIKE '"$lastday"%' and Archived <> 1;"
		/usr/local/bin/zmaudit.pl -y
		sleep 5
        freeamount=`df -m|grep "$partition"|cut -c 42-50|sed s/" "/""/g`
	                sleep 1
                done;

enjoy

victor diago

[zoneminder]

Hi Victor,

Thanks for this. I've just got one minor correction for you, my surname is actually 'Coombes' and not 'Combees'.

Other than that, everything is great!

Cheer,

Phil,

[victor_diago]

sorry for that. already fixed


could you put this in oficial release , creating a "contrib" directory.
it would be great

thanks

victor diago

[zoneminder]

Hi Victor,

I'll try and get a contrib folder in the next release.

Phil

[victor_diago]

i have made a little correction


put

Code: Select all

 freeamount=`df -m|grep "$partition"|cut -c 42-50|sed s/" "/""/g`

inside loop. so this way every day deleted the freeamount
variable will be refreshed.

i have already corrected it, so the code above is working great.

sorry for the error.
victor diago

[floyd_2]

Hi Victor. Nice script. I'll be using it tonite

You might consider making the following modification:

Original

Code: Select all

freeamount=`df -m|grep "$partition"|cut -c 42-50|sed s/" "/""/g`

Proposed

Code: Select all

freeamount=`df -m|grep "$partition"|awk '{ print $4 }'`

That way, if the column positions change for the free disk space value with a later patch etc (or the value is larger than allowed for by column position), you'll still get the right number.

Floyd

[floyd_2]

UPDATE - Bug in SQL repaired 15/8/2004, more logging added

Hi Victor...me again. Sorry to be modding your code again. Just a few bits and pieces to help it run a lil better.

1. I put in a counter in case the script goes into a loop. If the counter gets too high, the script will exit and log a message in the log file

2. Modified the SQL for $lastday so that it would definitely return the oldest date (was returning dates in random order). Also removed redundant code as a result of the sql mod

3. Moved the calculation of $lastday inside the main loop so that it could be recalculated for each loop iteration (otherwise it would look for and try to delete the same dates files recursively if more than 1 loop executed).

4. Added a logfile variable

5. Used awk in the $freeamount variable calculation per my last post.

Code: Select all

#!/bin/bash 
# This is the Second version of autoclean.sh for ZM Events. 
# autoclean.sh will remove (from filesystem AND database) 
# the oldest day, when your free disk amount is less than 
# the value defined in "lessdiskamount" 
# 
# See that This WILL NOT remove ARCHIVED events 
# so this is a very safe way to remove old events 
# 
# ZM (ZoneMinder) is the top Surveillance System in Linux 
# and its manteined by Philip Coombes. 
# 
# Fell Free to contact me (Victor Diago) in vdiago@spymac.com 
# 
# With a little help we could make ZM Always Better. 
# 
# Victor Diago 


# Put your partition (like hda1) where your events are. this is used to determine 
# available space. 
partition="your partition" 

# Put here your DataBase Name 
database="your db name (default zm)"

# Script will clean the oldest day when free space is less than XXX MB 

#lessdiskamount="less disk amount to allow (IN MB)" 
lessdiskamount="1500" 

# Your ROOT DB username 
rootdb="root s db username" 

# Your ROOT DB password 
rootpass="root s db password" 

# Location of log file 
logfile="/tmp/zm_cleanup.log" 

# Now we will calculate how much free space you have actually 

freeamount=`df -m|grep "$partition"|awk '{ print $4 }'` 

# Initialise a counter so we can be sure that the script wont loop forever 

let counter=0 
echo "`date` START Filesystem Freespace is ${freeamount} MB" >> $logfile

while [ "$freeamount" -lt "$lessdiskamount" ]; 
   do 
   # Now we need to calculate the oldest day recorded without Archive flag 

   lastday=`mysql -B -u "$rootdb" --password="$rootpass" -e "use zm;select min(StartTime) from Events where Archived <> 1;"| tail -1|cut -f1 -d " "` 

   # Increment the counter and check if the loop count is too high.  If high, exit and log a message 

   let counter=${counter}+1 

   if [ $counter -gt 20 ]; then 
      echo "`date` Possible loop detected. Exiting..." >> $logfile 
      exit 0 
   fi 

   # Remove events from DB and use zmaudit.pl to clean up the files on disk 

   mysql -B -u "$rootdb" --password="$rootpass" -e "use zm;delete from Events where StartTime LIKE '"$lastday"%' and Archived <> 1;" 
   /usr/local/bin/zmaudit.pl -y 

   sleep 10 
   freeamount=`df -m|grep "$partition"|awk '{ print $4 }'` 

   echo "`date` ***** Filesystem Cleanup Executed for $lastday. Freespace is $freeamount MB *****" >> $logfile

   sleep 1 
done;

echo "`date` END Filesystem Freespace is ${freeamount} MB" >> $logfile

[victor_diago]

great. VERY THANKS Floyd.

i dont know nothing about awk, so thanks for your contribution.

the counter is really a very good idea; i didnt thought about it because i run the software outside a loop. So it runs on my system every 10 minutes and, each time the script get the right values for free space, But VERY THANKS.

im very happy that you enjoyed it.

Im planning now (im with very few free time for these days)
to make a complete backup system for ZM, in CD, DVD, whatever. im thinking about make something that could have a feature to run in VCD / DVD video or something like that. so its something like : You select the camera to backup, then you select the period that you would like to save (eg since last week until now), then the software return how much free space you need to backup it, (CD OR DVD) then it asks if you want JPEG, MPEG or VCD format (i really think VCD format is a GREAT Thing because you could see the backup in any dvd home system). This for one camera only, for multiple camera the system should make something different. Im very excited about develop it.

I will have more free time next week, so next week i will make contact again with all yours that help to make zm always better.

Thank you all. (and very much to philip)

Victor Diago
Brazil

[floyd_2]

Sorry for the last version of the script. The line that determined the oldest date from the database didnt work correctly (mysql returned the query line as well as the single line of data requested). Have fixed this by tailing the output of the SQL (per Victor's original script) and added more useful logging.

I have modified the script in my previous post already by editing that post.

Floyd

[curtishall]

i have the latest version of this program...with the counter...I have:
lessdiskamount="1000"

everything else is okay (db settings and such). I have a cron script setup every hour to run....here's my problem:
Thu Nov 4 03:30:04 CST 2004 END Filesystem Freespace is 24091 MB
Thu Nov 4 04:30:04 CST 2004 START Filesystem Freespace is 16142 MB
Thu Nov 4 04:30:05 CST 2004 END Filesystem Freespace is 16142 MB
Thu Nov 4 05:30:04 CST 2004 START Filesystem Freespace is 8195 MB
Thu Nov 4 05:30:04 CST 2004 END Filesystem Freespace is 8195 MB
Thu Nov 4 06:30:01 CST 2004 START Filesystem Freespace is 835 MB

as soon as it reached below the lessdiskamount it freaked out.

now, the program is running however it decided to clean most of the events...so far:
/dev/sda2 117522304 70056212 47466092 60% /

and still going....what's the problem here?

[victor_diago]

Hi curtishall

So, lets see if i've issued your problem..

When less disk amount goes under your config (1000 MB) the autoclean DOESNT CLEAN WHAT IT SHOULD, right ?
Your partition config is ok ? is it set to sda2 or the true filesystem where you databases and files are ?

Victor Diago

[yosco]

Can you tell me more about how this should work? What happens after the limit is reached, How much does it take off the HD to prevent the system from going into a loop of cleaning the the data files. I tried to do a test and it seemed to also freak out. Is there any way to debug so I can see what it is doing? I know about the logs but it doesn't really provide any information.

[Renato]

I´m using FC3 and, when i running script, i receive the message :

autoclean2.sh: line 43: print: command not found

[yosco]

The function is built right into ZM, see this thread. No need to use these scripts if the filter works.

http://www.zoneminder.com/forums/viewto ... gewhenfull

[SyRenity]

The function is built right into ZM, see this thread. No need to use these scripts if the filter works.

http://www.zoneminder.com/forums/viewto ... gewhenfull

Yes, but the default built-in function does not save the archived events as well, correct?

CORRECTION: It will, as you can simply define a rule in the filter not to touch the archived events - which is set by default btw .