[SLUG] Tape Backups and Scripting
Please excuse my use of HTML formatting - it's the only way I could fit the script as written. I, probably like most, have only ever used tar for uncompressing files but have never actually used it with tape drives. We have a brand spanking new top of the range Dell server with 1.6Tb of disc drive and an Utrium 5 tape drive capable of holding 3Tb compressed (1.5Tb native). I have looked at Bacula (a pain to install on CentOS 5.6) and had a minor tilt at Amanda but would rather prefer to use command line scripting to run the backups as a cron job. Most things seem to go well but the backup does not seem to quite complete. You will see from the script below that I have listed a number of folders which I need to back up (literally the whole drive) with a few things like /tmp and /proc missing for obvious reasons. When I do a "list" of the files on the tape at the completion of the backup, it only ever displays as far as the "sys" folder with nothing after it (/var & /virtual). Why I wonder? I have also attempted to run a compare but my limited knowledge suggests that the line in my script is incorrect. A test restore of the /var folder to another partition reports that the data is not found so the backup is indeed incomplete. There is a bit of cutesy stuff in the script so that I can get a handle on times and so far, it takes exactly 3 hours to erase and almost exactly 1 hour to back up as far as it gets. This is not a server on the net so we can afford the luxury of shutting down some services in the middle of the night. This script would obviously be run as root. I am not a coder but have managed to cover all of my requirements over the years with BASH scripts. Suggestions from a luminary or two would be most welcome. Script follows: #!/bin/bash /bin/rm ~rick/Desktop/BackupCompleted.txt ## removes any previous file /bin/date '+START PROCESS: %H:%M:%S %d/%m/%y' > ~rick/Desktop/BackupCompleted.txt ## Time tagging file /bin/mt -f /dev/nst0 load /bin/mt -f /dev/nst0 rewind /bin/date '+START ERASE: %H:%M:%S %d/%m/%y' >> ~rick/Desktop/BackupCompleted.txt /bin/mt -f /dev/nst0 erase /bin/date '+COMPLETED ERASE: %H:%M:%S %d/%m/%y' >> ~rick/Desktop/BackupCompleted.txt /bin/date '+BACKUP COMMENCED: %H:%M:%S %d/%m/%y' >> ~rick/Desktop/BackupCompleted.txt /sbin/service mysqld stop /sbin/service httpd stop /bin/tar -clpzf /dev/nst0 /bin /boot /dev /etc /home /lib /lib64 /media /misc /mnt /net /opt /root /sbin /srv /sys /usr /var /virtual /bin/mt -f /dev/nst0 rewind /bin/date '+COMPLETED BACKUP: %H:%M:%S %d/%m/%y' >> ~rick/Desktop/BackupCompleted.txt /bin/date '+COMPARE COMMENCED: %H:%M:%S %d/%m/%y' >> ~rick/Desktop/BackupCompleted.txt /bin/tar -dlpzf /dev/nst0 /bin /boot /dev /etc /home /lib /lib64 /media /misc /mnt /net /opt /root /sbin /srv /sys /usr /var /virtual \ > ~rick/Desktop/CompareList.txt ## Coles out empty at the moment /bin/chown rick: ~rick/Desktop/CompareList.txt /sbin/service mysqld start /sbin/service httpd start /bin/mt -f /dev/nst0 offline /bin/date '+COMPLETED PROCESS: %H:%M:%S %d/%m/%y' >> ~rick/Desktop/BackupCompleted.txt /bin/chown rick: ~rick/Desktop/BackupCompleted.txt /bin/cat ~rick/Desktop/BackupCompleted.txt | mail -s Tape Backup Trial u...@someplace.net Oh, and the last line does not work but may be because of our very convoluted and protected mailing system but there may be a comment for someone to make there as well. The mail out would let me know that the job is complete.. Sendmail is running with an "out of the box" configuration. It has worked for me elsewhere this way. Thanks and with regards, Rick -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Tape Backups and Scripting
On Mon, Jun 06, 2011 at 06:25:29PM +1000, Rick Phillips wrote: > You will see from the > script below that I have listed a number of folders which I need to back > up (literally the whole drive) with a few things like /tmp and /proc > missing for obvious reasons. When I do a "list" of the files on the > tape at the completion of the backup, it only ever displays as far as > the "sys" folder with nothing after it (/var & /virtual). You need to exclude /sys for the same reason you exclude /proc - it is a filesystem constructed from kernel memory, not disk blocks. The list is maybe hanging in /sys because it's treating them like regular files (e.g. if you try to backup /dev/zero it will read zeroes forever and never complete). Look for an option to limit a backup within the specified filesystem, or not cross filesystem boundaries. In tar, the appropriate option is --one-file-system. In other system tools the option may also be called -x or -xdev. Oh, and using amanda or bacula will have benefits such as incremental dumps or multiple filesystems. Nick. -- PGP Key ID = 0x418487E7 http://www.nick-andrew.net/ PGP Key fingerprint = B3ED 6894 8E49 1770 C24A 67E3 6266 6EB9 4184 87E7 -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Tape Backups and Scripting
On 06/06/2011, at 6:25 PM, Rick Phillips wrote: > Please excuse my use of HTML formatting - it's the only way I could fit > the script as written. Same problem, so I attached it instead. > I, probably like most, have only ever used tar for uncompressing files > but have never actually used it with tape drives. As you've figured out, like a file, a tape is just a big sequence of bytes. > We have a brand spanking new top of the range Dell server with 1.6Tb of > disc drive and an Utrium 5 tape drive capable of holding 3Tb compressed > (1.5Tb native). Nice :) I've attached a rewritten, but logically similar script you can try...or not :) DISCLAIMER: it is untried - I don't have a tape-equipped Linux system to test on! I like scripting and your approach is a fairly common and relaxed way to run a backup. Not needing to do "hot" backups of MySQL is a luxury! Anyway, have a look at the script and feel free to contact me with any questions, or hit the list up for feedback. For the scripting gurus on the list, yes, I know there are many short-hand notations I can use to streamline things, but I wanted to keep it mostly readable while providing some portability/extensibility to the script. Cheers, James smime.p7s Description: S/MIME cryptographic signature -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] IPv6 using ufw on Debian stable
My hosting provider, Linode, now has IPv6 support as standard, so I thought I'd give it a go. The firewall script I've been using, firehol, seems to be unmaintained and doesn't support it, so I switched to ufw, which I've found to be excellent but despite supposedly supporting v6, I'm getting no data when it's enabled. I've enabled it in /etc/default/ufw: IPV6=yes Now when I try this with logging set to medium: ping6 ipv6.google.com I get this in the logs: Jun 7 11:56:23 stout kernel: [UFW ALLOW] IN= OUT=eth0 SRC=2600:3c00:::f03c:91ff:fe96:7edf DST=2001:4860:4002:0802::::1012 LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=25190 SEQ=1 Jun 7 11:56:23 stout kernel: [UFW AUDIT INVALID] IN=eth0 OUT= MAC=f2:3c:91:96:7e:df:c8:4c:75:f5:c4:ff:86:dd SRC=2001:4860:4002:0802::::1012 DST=2600:3c00:::f03c:91ff:fe96:7edf LEN=104 TC=0 HOPLIMIT=55 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=25190 SEQ=1 Jun 7 11:56:23 stout kernel: [UFW BLOCK] IN=eth0 OUT= MAC=f2:3c:91:96:7e:df:c8:4c:75:f5:c4:ff:86:dd SRC=2001:4860:4002:0802::::1012 DST=2600:3c00:::f03c:91ff:fe96:7edf LEN=104 TC=0 HOPLIMIT=55 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=25190 SEQ=1 Going there with a browser times out: lynx http://ipv6.google.com/ However if I disable the firewall, these things all work: simon@stout:~$ sudo ufw disable Firewall stopped and disabled on system startup simon@stout:~$ ping6 ipv6.google.com -c 3 PING ipv6.google.com(2001:4860:4002:802::1012) 56 data bytes 64 bytes from 2001:4860:4002:802::1012: icmp_seq=1 ttl=55 time=0.676 ms 64 bytes from 2001:4860:4002:802::1012: icmp_seq=2 ttl=55 time=0.719 ms 64 bytes from 2001:4860:4002:802::1012: icmp_seq=3 ttl=55 time=0.718 ms --- ipv6.google.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.676/0.704/0.719/0.029 ms Any thoughts on where I look next? The "INVALID" part of the logs just before BLOCK seems ominous, but I don't really know what it means. -- Simon Rumble -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] IPv6 using ufw on Debian stable
On Tue, Jun 07, 2011 at 12:03:50PM +1000, Simon Rumble wrote: > Jun 7 11:56:23 stout kernel: [UFW AUDIT INVALID] IN=eth0 OUT= > MAC=f2:3c:91:96:7e:df:c8:4c:75:f5:c4:ff:86:dd > SRC=2001:4860:4002:0802::::1012 > DST=2600:3c00:::f03c:91ff:fe96:7edf LEN=104 TC=0 HOPLIMIT=55 > FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=25190 SEQ=1 I guess it just means the firewall is blocking incoming ipv6 traffic. The firewall allowed your outbound packet through, but not the inbound response. Nick. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] IPv6 using ufw on Debian stable
On 7 June 2011 12:39, Nick Andrew wrote: > I guess it just means the firewall is blocking incoming ipv6 traffic. > The firewall allowed your outbound packet through, but not the inbound > response. > Yeah that's right. But not how it should work, according to what I've been reading. Here's the rules I have in place, which I should have included in the original email: simon@stout:~$ sudo ufw status verbose [sudo] password for simon: Status: active Logging: on (medium) Default: deny (incoming), allow (outgoing) New profiles: skip To Action From -- -- 22/tcp (OpenSSH) ALLOW INAnywhere 25/tcp (SMTP) ALLOW INAnywhere 80/tcp (WWW) ALLOW INAnywhere 25/tcp (Postfix) ALLOW INAnywhere 587/tcp (Postfix Submission) ALLOW INAnywhere 22/tcp (OpenSSH (v6)) ALLOW INAnywhere (v6) 25/tcp (SMTP (v6)) ALLOW INAnywhere (v6) 80/tcp (WWW (v6)) ALLOW INAnywhere (v6) 25/tcp (Postfix (v6)) ALLOW INAnywhere (v6) 587/tcp (Postfix Submission (v6)) ALLOW INAnywhere (v6) -- Simon Rumble -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] IPv6 using ufw on Debian stable
On 7 June 2011 12:47, Simon Rumble wrote: > On 7 June 2011 12:39, Nick Andrew wrote: > >> I guess it just means the firewall is blocking incoming ipv6 traffic. >> The firewall allowed your outbound packet through, but not the inbound >> response. >> > > Yeah that's right. But not how it should work, according to what I've been > reading. > You need a rule to allow packages for existing connections in - something like this: $IPT6 -A INPUT -i $PUB_IF -m state --state RELATED,ESTABLISHED -j ACCEPT I'm not sure if you need any special setup to tell to the kernel that connections should be tracked for v6 as well. I'm not familiar with ufw so I'm not sure how it's called there. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] IPv6 using ufw on Debian stable
On Tue, Jun 7, 2011 at 1:56 PM, miloska wrote: > You need a rule to allow packages for existing connections in - > something like this: > > $IPT6 -A INPUT -i $PUB_IF -m state --state RELATED,ESTABLISHED -j ACCEPT > > I'm not sure if you need any special setup to tell to the kernel that > connections should be tracked for v6 as well. > > I'm not familiar with ufw so I'm not sure how it's called there. I believe the old stable 2.6.18 kernel on linode doesn't support ipv6 connection tracking. If you're running that kernel, swapping to the new linode kernel will solve that. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
