Your message dated Fri, 09 Mar 2012 13:02:12 +0000
with message-id <[email protected]>
and subject line Bug#638920: fixed in backup-manager 0.7.10.1-1
has caused the Debian Bug report #638920,
regarding backup-manager: Please provide an uploaded files database
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
638920: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=638920
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: backup-manager
Version: 0.7.9-3
Severity: wishlist
Tags: upstream patch
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Currently when more than one job is run at the same day, backup-manager
uploads all archives of the day that exist in the repository in each job's
invocation. This results in extraneous uploads that consume bandwidth and
may also introduce errors later on, although the archive was succesfully
uploaded the first time.
It would be nice if backup-manager kept track of all succesfully uploaded
archives in a central database and consulted it to filter out unecessary
uploads.
The attached patch does exactly that, using a flat text file as the database.
In addition, other administrative tasks could be facilitated by such a
database (I, for example, use dar and isolated catalogs. If the database
exists I could make a cron job to remove archives that are uploaded and
eplace them with symlinks to the catalogs, to save space).
The patch has been tested and is used already on my system without producing
errors so far.
regards
George Zarkadas
- -- System Information:
Debian Release: 6.0.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'proposed-updates'), (500,
'stable'), (450, 'testing-proposed-updates'), (450, 'testing'), (400,
'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-5-amd64 (SMP w/4 CPU cores)
Locale: LANG=el_GR.utf8, LC_CTYPE=el_GR.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages backup-manager depends on:
ii debconf [debconf-2.0] 1.5.36.1 Debian configuration management sy
ii findutils 4.4.2-1+b1 utilities for finding files--find,
ii ucf 3.0025+nmu1 Update Configuration File: preserv
backup-manager recommends no packages.
Versions of packages backup-manager suggests:
ii anacron 2.3-14 cron-like program that doesn't go
ii backup-manager-doc 0.7.9-3 documentation package for Backup M
ii dar 2.3.10-1+b1 Disk ARchive: Backup directory tre
ii dvd+rw-tools 7.1-6 DVD+-RW/R tools
ii genisoimage 9:1.1.11-1 Creates ISO-9660 CD-ROM filesystem
ii gettext-base 0.18.1.1-3 GNU Internationalization utilities
ii libfile-slurp-perl 9999.13-1 single call read & write file rout
pn libnet-amazon-s3-perl <none> (no description available)
ii openssh-client 1:5.5p1-6+squeeze1 secure shell (SSH) client, for sec
ii perl 5.10.1-17squeeze2 Larry Wall's Practical Extraction
ii wodim 9:1.1.11-1 command line CD/DVD writing tool
ii zip 3.0-3 Archiver for .zip files
- -- debconf information excluded
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBAgAGBQJOUt2XAAoJEJWXIVmJ5BwWHQ8IALobsYprBJu5FVBwcVOXQGkB
EmCgOxVd5ogiZQ2VVxKAQ9F5HBhX/JyjRxI3jxbc2gq3Dfn2FsyQFXpM6rGs9eIu
3yGTAIY1YtV2bHfeiUnl9hhQF3RQcuJ1nClLTw8cifWuGb+3qbmAWcSBicbJmYHV
KhDFncSiKHG7v4Q0uvGhMufAK31uJQQVachLHlex/9fTcZBPatHwFP39Jr6ZwZxA
j5/jwZx9SRDFAnlbOlHwfWjIYxdzrC5GLk95iF5rDYNv/J2+IUQMu6m6vywsskzI
kFXhDQoH9+Ki/rx+BvJ+unQK0TMXAbxyNwYH0gB7T22FJ4v+NoPhrGlCum5M/5E=
=oUkf
-----END PGP SIGNATURE-----
--- a/backup-manager-upload
+++ b/backup-manager-upload
@@ -105,6 +105,61 @@
}
}
+# The idea behind BM_UPLOADED_ARCHIVES is to have a database of what archives
+# have been uploaded so far. This allows multiple execution of upload actions
+# within a day without resending all archives of the day from the beginning.
+
+# Add one file,host pair to $BM_UPLOADED_ARCHIVES database.
+# Called immediately *after* successful uploading of an archive.
+sub appendto_uploaded_archives($$)
+{
+ my $file = shift;
+ my $host = shift;
+ unless ( defined $file and defined $host ) {
+ print_error "required args needed";
+ return FALSE;
+ }
+
+ my $upload_fname = $ENV{BM_UPLOADED_ARCHIVES};
+ unless ( defined $upload_fname ) {
+ # Uncomment next line if you want the mandatory use
+ # of BM_UPLOADED_ARCHIVES (ie always have it around).
+ #print_error "BM_UPLOADED_ARCHIVES is not defined";
+ return FALSE;
+ }
+
+ # if $file already in database, append host to that line;
+ # else append a lines "$file $host" to the end.
+
+ my $io_error = 0;
+ if ( ! system( "grep -q \"^$file \" $upload_fname" ) ) {
+ my $cmd = "sed -i \"s:^$file .*\$:\& $host:\" $upload_fname";
+ $io_error = system("$cmd");
+ }
+ elsif ( open(my $fh, ">>", $upload_fname) ) {
+ print($fh "$file $host\n") or $io_error = 1;
+ close $fh;
+ }
+ else {
+ $io_error = 2;
+ }
+ if ( $io_error ) {
+ print_error "IO error: did not update $upload_fname with '$file
$host'";
+ return FALSE;
+ }
+
+ return TRUE;
+}
+
+# Get all files of the specified date; filter the list through
+# BM_UPLOADED_ARCHIVES if it is set in the environment.
+# NOTE: Doing the filtering here implies that the archive is considered
+# uploaded if a single upload to a host succeeds; that is even when there
+# are failures to other hosts (in case of multiple host uploading).
+# To consider it uploaded when all hosts succeed, the filtering must be
+# transfered to the individual upload subroutines (and check for existence
+# of file,host pair in the database).
+#
sub get_files_list_from_date($)
{
my $date = shift;
@@ -129,8 +184,21 @@
exit E_INVALID;
}
- while (<$g_root_dir/*$date*>) {
- push @{$ra_files}, $_;
+ my $upload_fname = $ENV{BM_UPLOADED_ARCHIVES};
+ if ( defined $upload_fname ) {
+ # filter file list through the BM_UPLOADED_ARCHIVES database
+ while (<$g_root_dir/*$date*>) {
+ my $file = $_;
+ my $cmd = "grep -q '$file' $upload_fname";
+ if ( system ("$cmd") ) {
+ push @{$ra_files}, $file;
+ }
+ }
+ }
+ else {
+ while (<$g_root_dir/*$date*>) {
+ push @{$ra_files}, $_;
+ }
}
return $ra_files;
@@ -267,7 +335,12 @@
print_error ("Unable to upload \"$file\". ".($! || $@ || $ret));
return 0;
}
- return 1;
+ else {
+ # use same name in both cases (gpg encryption is done on the fly);
+ # continue if writing to uploaded archives file fails.
+ appendto_uploaded_archives($file, $host);
+ }
+ return 1;
}
# How to upload files with scp.
@@ -551,17 +624,24 @@
# Put all the files over the connexion
foreach my $file (@{$ra_files}) {
chomp $file;
+ # continue if writing to uploaded archives file fails.
if ($BM_UPLOAD_FTP_SECURE) {
- unless (ftptls_put_file ($ftp, $file)) {
- print_error "Unable to transfer $file";
- return FALSE;
- }
+ if (ftptls_put_file ($ftp, $file)) {
+ appendto_uploaded_archives($file, $host);
+ }
+ else {
+ print_error "Unable to transfer $file";
+ return FALSE;
+ }
}
else {
- unless (ftp_put_file ($ftp, $file)) {
- print_error "Unable to transfer $file: " . $ftp->message;
- return FALSE;
- }
+ if (ftp_put_file ($ftp, $file)) {
+ appendto_uploaded_archives($file, $host);
+ }
+ else {
+ print_error "Unable to transfer $file: " . $ftp->message;
+ return FALSE;
+ }
}
}
print_info "All transfers done, loging out from $host\n";
@@ -727,6 +807,8 @@
);
$uploaded{$filename} = $file_length;
}
+ # For the S3 method, we assume success in any case.
+ appendto_uploaded_archives($file, $host);
}
# get a list of files and confirm uploads
--- a/backup-manager.conf.tpl
+++ b/backup-manager.conf.tpl
@@ -305,6 +305,20 @@
# Where to put archives on the remote hosts (global)
export BM_UPLOAD_DESTINATION=""
+# Uncomment the 'export ...' line below to activate the uploaded archives
+# database.
+# Using the database will avoid extraneous uploads to remote hosts in the
+# case of running more than one backup-manager jobs per day (such as when
+# you are using different configuration files for different parts of your
+# filesystem).
+# Note that when you upload to multiple hosts, a single succesfull upload
+# will mark the archive as uploaded. Thus upload errors to specific hosts
+# will have to be resolved manually.
+# You can specify any filename, but it is recommended to keep the database
+# inside the archive repository. The variable's value has been preset to
+# that.
+#export
BM_UPLOADED_ARCHIVES=${BM_REPOSITORY_ROOT}/${BM_ARCHIVE_PREFIX}-uploaded.list
+
##############################################################
# The SSH method
#############################################################
--- End Message ---
--- Begin Message ---
Source: backup-manager
Source-Version: 0.7.10.1-1
We believe that the bug you reported is fixed in the latest version of
backup-manager, which is due to be installed in the Debian FTP archive:
backup-manager-doc_0.7.10.1-1_all.deb
to main/b/backup-manager/backup-manager-doc_0.7.10.1-1_all.deb
backup-manager_0.7.10.1-1.debian.tar.gz
to main/b/backup-manager/backup-manager_0.7.10.1-1.debian.tar.gz
backup-manager_0.7.10.1-1.dsc
to main/b/backup-manager/backup-manager_0.7.10.1-1.dsc
backup-manager_0.7.10.1-1_all.deb
to main/b/backup-manager/backup-manager_0.7.10.1-1_all.deb
backup-manager_0.7.10.1.orig.tar.gz
to main/b/backup-manager/backup-manager_0.7.10.1.orig.tar.gz
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Sven Joachim <[email protected]> (supplier of updated backup-manager package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Wed, 07 Mar 2012 19:08:15 +0100
Source: backup-manager
Binary: backup-manager backup-manager-doc
Architecture: source all
Version: 0.7.10.1-1
Distribution: unstable
Urgency: low
Maintainer: Sven Joachim <[email protected]>
Changed-By: Sven Joachim <[email protected]>
Description:
backup-manager - command-line backup tool
backup-manager-doc - documentation package for Backup Manager
Closes: 491199 567553 602633 608237 610413 637743 638803 638920 655985 659162
Changes:
backup-manager (0.7.10.1-1) unstable; urgency=low
.
[ Sven Joachim ]
* New upstream release (Closes: #655985).
- Adds a new "postgresql" backup method (Closes: #637743).
* Remove the fragile translation-updates.diff patch. Instead, backup
the upstream po files before the build and restore them in the clean
target (Closes: #602633).
* Depend on perl (via ${perl:Depends}), rather than than suggesting it,
since perl is required by backup-manager-purge (LP: #941919).
* Drop the debian-perl-location.diff patch, setting PERL5DIR in
debian/rules instead.
* Refresh the remaining Debian patches.
* Fill out the same "Language:" header in debian/po/*.po.
* Drop findutils dependency, even the version in Etch is new enough.
* Upgrade Standards-Version to 3.9.3, no changes needed.
* Update lintian override for Lintian 2.5.
* Update debian/watch to find the 0.7.10.1 release tarball.
* Add build-indep and (empty) build-arch targets to debian/rules.
* Update Vcs-* fields in debian/control, switching to collab-maint.
* Fix a few typos in the debconf templates, thanks to Daniele Forsi
for the report.
* Update README.Debian, the Amazon S3 upload method not longer needs
libfile-slurp-perl (see #491199).
* Demote the priority of the repo_user/repo_group debconf templates to
medium.
* Update debian/copyright.
.
[ Georgios M. Zarkadas ]
* Add Greek debconf translation.
* Allow absolute paths in BM_TARBALL_BLACKLIST even when backing up
the root (/) directory (Closes: #608237).
* Fix missed archive uploads when backups are crossing the day
limit (Closes: # 638803).
* Provide a configuration option to record all uploaded archives in
a single text database file (Closes: #638920).
* Fix upstream makefile to make the build/clean process of gettext (po)
files idempotent and bring sanity back to the package's build process.
* Fix a security issue in pgsql backup method regarding the .pgpass
file handling.
* Add support for local unix sockets connections to the pgsql backup method.
* Add existence check for pg_dumpall to the pgsql backup method.
* Use the same naming convention for database methods generated archives.
* Add myself to Uploaders.
* Add cron script to remove old entries from uploaded archives database.
* Update control file to reflect upstream fix of Amazon S3 uploading
method's dependencies (Closes: #491199).
* Fix missed warnings when BM_LOGGER* variables are undefined or wrongly
configured (Closes: #567553).
* Change configurator to dynamically update BM_TARBALL_BLACKLIST when
user selections change during configuration (Closes: #610413).
* Add backtracking support to the configurator.
.
[ Sven Joachim, Georgios M. Zarkadas ]
* Switch source format to 3.0 (quilt).
- Drop debian/README.source.
* Change debian/rules to use the generic 'dh$@' rule plus overrides.
.
[ Joe Dalton ]
* Add Danish debconf translation (Closes: #659162).
Checksums-Sha1:
52102a178a9a5c99d031b2be7f1a756f7c441d90 2193 backup-manager_0.7.10.1-1.dsc
a08768acff444275c3b33e35a808280106060641 150963
backup-manager_0.7.10.1.orig.tar.gz
e54e17f9a6ff9a78bb983f6d9f1653f1f9ac19b8 75326
backup-manager_0.7.10.1-1.debian.tar.gz
b959bc66c6b042ce36240bc3f9a2dc0e367fca3f 159432
backup-manager_0.7.10.1-1_all.deb
db61174d7f10f7a06d7e5c90c6e29772486523b1 234504
backup-manager-doc_0.7.10.1-1_all.deb
Checksums-Sha256:
bcf6a886a56bf649734293a6729642b762cbfa4d5cebb3ab56892d2a34561995 2193
backup-manager_0.7.10.1-1.dsc
16994520d3fded41e166d17c946405dd192acf960603044aa6c915465e78d41f 150963
backup-manager_0.7.10.1.orig.tar.gz
46b68be7b01a990c5e9a869d7fcf7d2f7ae82d5dd614210d66f5342db45faaac 75326
backup-manager_0.7.10.1-1.debian.tar.gz
b701261e84675312ba11b9a6204698b42be333c4ba31b41cd50728bf64a60621 159432
backup-manager_0.7.10.1-1_all.deb
f09760ba85ad0716f4ca37e20c630a76db5912f78779a45d673a80de8dbaf323 234504
backup-manager-doc_0.7.10.1-1_all.deb
Files:
e1e1cbd3b3ed95849724296c81bbc390 2193 admin optional
backup-manager_0.7.10.1-1.dsc
b7f1e6ecfb6c45116e9aef2de92ffb12 150963 admin optional
backup-manager_0.7.10.1.orig.tar.gz
6c36718a802ebea270f79a162a9dab10 75326 admin optional
backup-manager_0.7.10.1-1.debian.tar.gz
626a3bd130e39ac359c0d98240026396 159432 admin optional
backup-manager_0.7.10.1-1_all.deb
6efe622a55bb99d2701966eb2e23d5b8 234504 doc optional
backup-manager-doc_0.7.10.1-1_all.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iQIcBAEBCAAGBQJPWf2oAAoJEMcZdpmymyMqGWEQAJ+XQDnWcjnvTlPA7U1TQpOz
L4H45lB5N0w9pkxORnxw6qG9KU1FZFtdtiFUNqEsNFYdN4h6FI/NL9Un5xMh6Zhb
ApMKftW5BVdLaXqW0Wr+y0+ejeeZDmr/3DACGqQgKFEelna6It2bp4UeNQEqSRdZ
EH2LldQNCw5nhsJt9WDoXofyjTPA3jiD9BpYal9PXoRKWuFRTvWBB1GTm5mi5LCB
M92IhvELpd1TEG3eC4OHLI1gaQsHe9n6G2sPBSU7VGbSkCUBfjXBG4iAUPAcQO+G
eFAt0Yfml/6o76Q762OJN27/0c7LmwS3WwzhDGYbpL1nWIBiscbK2Aw2CzrtY58r
IOw0nLOUolj04PmfF9Mgs39EVrxxr4teix3hyBuxnuga2Ftih/NH3hT8Zb9Y+TA4
umeSKu0DRCNuRAMfaGS+iKaqSnpTFlp4P1hynNkPcVptY6fhqfpGuOLHZrkr9kmF
IK8Tue561mFQNUqx/AqOiNMs46xlDy7kci/zrLIUhjKnZ7QMEtiNTqqewz6YUZfD
jZWHioJIFSHTLCuiILDdFGrW01o9a/Rt/kNJmn6xCJfQ3j1iuKEwVWSWKOy1JQo/
sUL5SJzuPKl3WKYvy7DOFKd5JIOVMLubtNpeg5LMoEBs/Tobmuj0hbjOAtAkwzvW
WOjEZGO/1zC7FbWL+ZlE
=OHCS
-----END PGP SIGNATURE-----
--- End Message ---
