@@ -32,7 +32,7 @@
date = calendar.datetime.date.today()
suffix = str( date.month) + '-' + str(date.day) + '-' + str(date.year)
exList = '/tmp/excludes_%s.txt'%suffix
- f = open(exList,'w')
+ f = open(exList,'a')
for root, dirs, files in os.walk( directory ):
# Keep count of files meeting EXTENTIONS criteria
@@ -55,7 +55,7 @@
# We never want to add the top level directory to exclusion
# and want to add directory to exclusion if no files matching
# EXTENTION criteria found
- if ( ( count == 0 ) and ( root != directory ) ):
+ if ( ( count == 0 ) and ( root != directory ) and ( len( dirs ) == 0 ) ):
#f.write( root.split( directory , 1 )[1] + '\n' )
f.write( '.' + root.split( directory , 1 )[1] + '\n' )
Thanks,
Vijay Avarachen
On 2/18/06, Vijay Avarachen <[EMAIL PROTECTED]> wrote:
Craig,
I finally got it working. I figured out the issues too. I was making three mistakes:
[1] Using -P switch for absolute path
[2] Not switching into the directory (-C)
[3] Did not specify the '.' at the end of the tar command. Which basically means switch to this directory (-C) and back current directory up.
I also updated the exclusion.py script to conform to the changes made to backup.sh.
Config.pl
$Conf{XferMethod} = 'tar';
$Conf{TarShareName} = ['/working1' , '/working2' , '/working3'];
$Conf{TarClientCmd} = '$sshPath -q -x -n -l root $host nice -n 21 sh /root/scripts/BackupPC/backup.sh $shareName+';
backup.sh
#!/bin/sh
rm -f /tmp/excludes_*
/usr/bin/python /root/scripts/BackupPC/exclusion.py
tar -c -v -p -f - -C $1 -X /tmp/excludes_* --totals .
exclusion.py
#!/usr/bin/env python
'''
The goal of this script is to create an exclusion list for the tar
executable. This script adds files to exclusion list based on following
parameters:
[1] If a files extention does NOT match EXTENTIONS dictionary
[2] If a file does NOT have an extention
[3] If a directory does NOT contain any instances of files of EXTENTIONS
The exclusion list is created in /tmp/exclusion_DATE
TO DO:
[1] DIRROOT and EXTENTION as ARGS
[2] Try-catch blocks
'''
import os,calendar
from os.path import join, getsize
## List of directories to be processed
DIRROOT = [ '/working1' , '/working2' , '/working3' ]
## Dictionary containing extentions to watch for. Each DIRROOT can have its own list
EXTENTIONS = { '/working1':['inp'] }
def BuildExcludeList( directory , extention ):
''' Creates an exclude list for tar command '''
date = calendar.datetime.date.today()
suffix = str(date.month) + '-' + str(date.day) + '-' + str(date.year)
exList = '/tmp/excludes_%s.txt'%suffix
f = open(exList,'w')
for root, dirs, files in os.walk( directory ):
# Keep count of files meeting EXTENTIONS criteria
count = 0
for name in files:
if name.count('.') >= 1:
# text after the last period assumed to be extention
ext = name.split('.')[-1].lower()
# Check to see if ext is in EXTENTIONS
if extention.count( ext ) == 0:
# Add file path to exclusion list (minus rood dir)
f.write( '.' + join( root , name ).split( directory , 1 )[1] + '\n' )
else:
# Match found. This dir will not be added to exclusion
count += 1
else:
# File has no extention, add to exclusion
f.write( '.' + join( root , name ).split( directory , 1 )[1] + '\n' )
# We never want to add the top level directory to exclusion
# and want to add directory to exclusion if no files matching
# EXTENTION criteria found
if ( ( count == 0 ) and ( root != directory ) ):
f.write( '.' + root.split( directory , 1 )[1] + '\n' )
# Skip subversion directories
if '.svn' in dirs:
dirs.remove( '.svn' )
f.close()
if __name__ == '__main__':
for D in DIRROOT:
for E in EXTENTIONS[D]:
BuildExcludeList( D , E )
Thanks for all your help. Next im gonna try to get the incremental and restores working.
Vijay AvarachenOn 2/18/06, Vijay Avarachen < [EMAIL PROTECTED] > wrote:Craig,
I wanted to improve the file exclusion, so i a small python script. It works perfectly. However the backs are still failing with the same error :-(
I am absolutely certain that the python script does not output anything on the screen.
Right now I am calling the python script from the backup.sh script...ill change this later.
This is my backup.sh
#!/bin/sh
rm -f /tmp/excludes_*
/usr/bin/python /root/exclusion.py
tar -c -v -p -P -f - -X /tmp/excludes_* $1 --totals
Here is my exclusion.py. This basically creates a exclusion file called /tmp/exclusion_<date>.txt which gets passed to tar with --exclude by back.sh
<deleted>
By the way when I run the backup.sh script manually without the '-' (don't pipe the output), this works perfectly.
tar -c -v -p -P -f /tmp/test.tar $1 -X /tmp/excludes_* --totals
But when I am pipeing the output via ssh, the backup seem to fail :-( I also tried to place the -X option after $1
Here is the output of ssh -q -x -n -l root HOST nice -n 21 sh /root/backup.sh /working1 | tar tvf - per your suggestion
-bash-3.00$ ssh -q -x -n -l root spduslisfea07 sh /root/backup.sh /working1 | tar tvf -
/working1/
/working1/engine-gasket/
/working1/engine-gasket/all3-fromhm-catenated.inp
drwxrwxr-x alex.he/SPDUSLIS-FEA 0 2006-02-10 13:37:28 /working1/
drwxr-x--- root/root 0 2006-02-17 19:50:13 /working1/engine-gasket/
-rw-rw---- alex.he/SPDUSLIS-FEA 87376746 2005-06-27 15:18:28 /working1/engine-gasket/all3-fromhm-catenated.inp
/working1/engine-gasket/engine-fromhm-untrans.inp
-rw-rw---- alex.he/SPDUSLIS-FEA 9140737 2005-06-27 15:20:04 /working1/engine-gasket/engine-fromhm-untrans.inp
/working1/engine-gasket/mysymlink.inp
/working1/engine-gasket/nmap.inp
/working1/engine-gasket/restart.inp
/working1/engine-gasket/restart2.inp
Total bytes written: 96532480 (93MiB, 12MiB/s)
lrwxrwxrwx root/root 0 2006-02-17 19:50:13 /working1/engine-gasket/mysymlink.inp -> nmap.inp
-rw-rw---- alex.he/SPDUSLIS-FEA 360 2005-06-27 15:19:02 /working1/engine-gasket/nmap.inp
-rw-rw---- alex.he/SPDUSLIS-FEA 2408 2005-06-27 15:16:46 /working1/engine-gasket/restart.inp
-rw-rw---- alex.he/SPDUSLIS-FEA 1275 2005-06-27 15:16:36 /working1/engine-gasket/restart2.inp
PS. Sorry I forgot to sent the last email to [EMAIL PROTECTED] orge.net
Thanks for your help,
Vijay AvarachenOn 2/17/06, Craig Barratt < [EMAIL PROTECTED]> wrote:"Vijay Avarachen" writes:
> I don't understand why tar does not support an include parameter.
>
> Anyways here is what I did and it kindda seems to work:
> in the config.pl in the <hostname> folder I have:
> $Conf{XferMethod} = 'tar';
> $Conf{TarShareName} = ['/working1'];
> $Conf{TarClientCmd} = '$sshPath -q -x -n -l root $host nice -n 21 sh
> /root/backup.sh $shareName+';
>
> On the host to be backed up (client) I have the following in
> /root/backup.sh:
> #!/bin/sh
> find $1 -type f | egrep -vi *.inp > /tmp/backup_`date '+%d-%m-%Y'`
> tar -c -v -p -P -f - $1 -X /tmp/backup_`date '+%d-%m-%Y'` --totals
You have to be very sure that this command generates no output
prior to the tar archive. Are you sure there are no extraneous
messages from /bin/sh, find etc?
Other comments:
- do you need to quote "*.inp"?
- if find runs before midnight and tar after midnight, then
`date '+%d-%m-%Y'` will evaluate to different strings.
- I'm not sure if the -X option should be prior to $1.
I'd recommend running the command manually and looking at the
output with, eg, od -c:
ssh -q -x -n -l root HOST nice -n 21 sh /root/backup.sh /working1 | od -c
or
ssh -q -x -n -l root HOST nice -n 21 sh /root/backup.sh /working1 | tar tvf -
> Also the directory I am trying to backup is /working1, but when I go to
> browse the backups, I see /working1/working1/<other folders>. I don't know
> why /working1 is repeated twice. Any pointers would be appreciated.
You should cd to $1, and pass "." to tar, since the archive should
be relative to the share name, not including the sharename:
cd $1
tar -c -v -p -P -f - . -X /tmp/backup_`date '+%d-%m-%Y'` --totals
Craig--
"Knowledge is the only wealth that grows as you spend it, and diminishes as you save it."
-- ancient Sanskrit saying
--
"Knowledge is the only wealth that grows as you spend it, and diminishes as you save it."
-- ancient Sanskrit saying
--
"Knowledge is the only wealth that grows as you spend it, and diminishes as you save it."
-- ancient Sanskrit saying
