If anyone has tried using the ftp upload functionality you will probably realise it can generate quite a lot of data, therefore requiring a large ftp site.
I have been unable to find a reasonably priced FTP server, so am stuck with the 10M of free webspace space I get with my ISP.
In order to get around this I have developed some scripts to automatically clear out enough old events from the FTP server before uploading the next event. I've been using it for the last few days and it seems to be working OK.
This should mean that if your PC is stolen, you should have the last x events on your FTP server. Whilst ideally you would have all events, Hopefully one of these last events should be the one you need to identify the culprit.
Apologies in advance for not writing this in perl and integrating with ZM, but I simply dont know perl.
If you want to use this script please do so, but obviously it is entirely at your own risk. It has only been tested on SUSE 10 - other distros may give different results.
INSTALLING AND CONFIG
Firstly, get FTP uploading working before you do anything else.
Hopefully the formatting of these scripts will not be screwed up by pasting them into this forum. If so I can email them to people instead.
Then create a zmftplist.sh file in /usr/local/bin and paste in the following:
------------------------------------------------------------------------
#!/bin/sh
FTPSERVER=yourftpserver
USERNAME=yourlogin
PASSWORD=yourpassword
FTPDIR=/the directory on your FTP server to use for events
ftp -n $FTPSERVER <<!
quote user $USERNAME
quote pass $PASSWORD
binary
cd $FTPDIR
ls -ltr
quit
!
-------------------------------------------------------------------------
Change it to be owned by root, with all execute permissions.
Ensure the variables are amended to be the same as set in zoneminder.
Check that it runs and displays a list of files in your ftp server.
Once this is running, do the same for another file, /usr/local/bin/zmftpclear.sh
----------------------------------------------------------------------------
#!/bin/bash
# Pass the filename required as first parameter
FTPSPACE=9000000 # Set to the amount of FTP space available
FTPSERVER=yourftpserver
USERNAME=yourlogin
PASSWORD=yourpassword
FTPDIR=/the directory on your FTP server to store events
date
date >> /tmp/zmftpclear.log
echo Clearing enough space for $1 >> /tmp/zmftpclear.log
required=`ls -l $1 | awk {'print $5'}`
echo "Bytes required is $required" >> /tmp/zmftpclear.log
# Generate a list of current files
/usr/local/bin/zmftplist.sh | grep "^-" | grep ":" | awk {'print $5" "$9'} > /tmp/zmftpfilelist.txt
# Calculate the total size of files we currently have
# And store file details in an array
x=0
total=0
declare -a FILEARRAY # Array to hold file details
declare -a REMOVELIST # Array to hold files to be deleted
recordtype=bytes
cleared=0
# Scan through the listing and store in array
for all_files in `cat /tmp/zmftpfilelist.txt`
do
if [ $recordtype == bytes ] # Store the bytes in even array position
then
total=$(($total+$all_files))
FILEARRAY[$x]=$all_files
# echo bytes array ${FILEARRAY[$x]}
recordtype=names
x=$((x+1))
else # and the filenames in odd array positions
FILEARRAY[$x]=$all_files
# echo filename ${FILEARRAY[$x]}
recordtype=bytes
x=$((x+1))
fi
done
xmax=$((x))
x=0
# We now have the total bytes used
echo Total Bytes used is $total >> /tmp/zmftpclear.log
total=$((total+$required))
echo Total Bytes needed to also fit file is $total >> /tmp/zmftpclear.log
# and can work out the number of bytes free, if any
maxfree=$(($FTPSPACE-$total))
echo Total Free is $maxfree >> /tmp/zmftpclear.log
echo Now working out if we need to clear space
# If the required space is greater than free space, then we need to clear files down to make room
# if [ $maxfree -gt $required ]
if [ $maxfree -gt 1 ]; then
echo No problem - space available for $1 >> /tmp/zmftpclear.log
elif [ $required -gt $FTPSPACE ]; then
echo Nothing we can do - cant fit $required into $FTPSPACE whatever we do >> /tmp/zmftpclear.log
else
echo Need to clear some space
echo First generating the ftp script to run
# Clear the existing script
cat /dev/null > /tmp/ftpdel.sh
echo "ftp -nv $FTPSERVER <<!" >> /tmp/ftpdel.sh
echo "quote user $USERNAME" >> /tmp/ftpdel.sh
echo "quote pass $PASSWORD" >> /tmp/ftpdel.sh
echo "cd $FTPDIR" >> /tmp/ftpdel.sh
# Calculate how much we need to clear out
toclear=$((0-$maxfree))
echo Clearing down $toclear bytes for $1 >> /tmp/zmftpclear.log
# echo " " >> /tmp/zmftpclear.log
echo FILENAME SIZE CLEARED TO CLEAR >> /tmp/zmftpclear.log
while [ $cleared -lt $toclear ] # Keep adding delete statements until enough room
do
echo "del ${FILEARRAY[(($x+1))]}" >> /tmp/ftpdel.sh
cleared=$((cleared+FILEARRAY[$x]))
echo ${FILEARRAY[$(($x+1))]} ${FILEARRAY[$x]} $cleared $(($toclear-$cleared)) >> /tmp/zmftpclear.log
x=$(($x+2))
done
# Finish off FTP script
echo "quit" >> /tmp/ftpdel.sh
echo "!" >> /tmp/ftpdel.sh
# Now run the script
chmod 750 /tmp/ftpdel.sh
/tmp/ftpdel.sh
chmod a-x /tmp/ftpdel.sh
echo Completed >> /tmp/zmftpclear.log
fi
echo " " >> /tmp/zmftpclear.log
-----------------------------------------------
Change it to be owned by root, with all execute permissions.
Amend the variables again at the top, setting FTPSPACE to the amount of space in bytes you want to allocate to events - in my case about 9M - 9000000.
Now you need to integrate the shell scripts into ZM.
Make a copy of /usr/local/bin/zmfilter.pl
Search for the line:
$ftp->cwd( ZM_UPLOAD_FTP_REM_DIR ) or warn( "FTP - Can't cwd" );
and add in the following 6 lines of code in between:
-----------------------------------------------------------------------------------------
$ftp->login( ZM_UPLOAD_FTP_USER, ZM_UPLOAD_FTP_PASS ) or warn( "FTP - Can't login" );
$ftp->binary() or warn( "FTP - Can't go binary" );
$ftp->cwd( ZM_UPLOAD_FTP_REM_DIR ) or warn( "FTP - Can't cwd" );
# Just before we put, lets clear space
# Added by K.Robson as part of zmftpclear
$ENV{PATH} = "/bin:/usr/bin";
delete @ENV{ 'IFS', 'CDPATH', 'ENV', 'BASH_ENV' };
my $command = "/usr/local/bin/zmftpclear.sh $arch_file";
system($command);
$ftp->put( $arch_file ) or warn( "FTP - Can't upload '$arch_file'" );
$ftp->quit() or warn( "FTP - Can't quit" );
unlink( $arch_file );
---------------------------------------------------------------------------------------------
This will cause our script to run before the file is ftped.
All things being well you should get additional info in the /tmp/zmfilter.log file, and a new file /tmp/zmftpclear.log
The zmfilter.log file should show whether space needs to be allocated on the server:
Now working out if we need to clear space
Need to clear some space
The /tmp/zmftpclear.log file should give more detailed info:
Sun Mar 19 17:10:19 GMT 2006
Clearing enough space for /tmp/Drive-10368.zip
Bytes required is 1459259
Total Bytes used is 8758511
Total Bytes needed to also fit file is 10217770
Total Free is -1217770
Clearing down 1217770 bytes for /tmp/Drive-10368.zip
FILENAME SIZE CLEARED TO CLEAR
Drive-10361.zip 2019104 2019104 -801334
Completed
The above shows that 10368.zip was 1459259 bytes in size.
Currently 8758511 bytes were used, so 1217770 bytes needed to be deleted to keep the total after upload to under 9M.
To do this, file 10361.zip - the oldest file - was deleted from the server. Depending on space, multiple files may be deleted.
You may also get:
Sun Mar 19 16:38:18 GMT 2006
Clearing enough space for /tmp/Drive-10363.zip
Bytes required is 1891555
Total Bytes used is 6240235
Total Bytes needed to also fit file is 8131790
Total Free is 868210
No problem - space available for /tmp/Drive-10363.zip
Which means the file could fit without any problems so nothing was done.
You may also get:
Sun Mar 19 08:13:13 GMT 2006
Clearing enough space for /tmp/Drive-10304.zip
Bytes required is 13006478
Total Bytes used is 0
Total Bytes needed to also fit file is 13006478
Total Free is -3006478
Nothing we can do - cant fit 13006478 into 10000000 whatever we do
Which means the file itself is bigger than the allowable space, so we just go ahead and let it fail.
Thats it. Hopefully it will work without problems. If not then work your way through each bit of the script yourself.
1. Check FTP uploading is working before anything else.
2. Start with checking zmftplist.sh works
3. Then run zmftpclear.sh runs manually - remember to change permissions on any log files generated so that your ZM/Web user can write to them when it runs.
Please update this forum entry if you find any problems with it. As I said its been running for a few days without problems, but it doesn't mean I've thought of everything. Apologies in advance, but I wont be able to respond for a few weeks.
FTP Server Maintainance
-
zoneminder
- Site Admin
- Posts: 5215
- Joined: Wed Jul 09, 2003 2:07 pm
- Location: Bristol, UK
- Contact: