Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package borgmatic for openSUSE:Factory checked in at 2022-03-21 20:11:42 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/borgmatic (Old) and /work/SRC/openSUSE:Factory/.borgmatic.new.25692 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "borgmatic" Mon Mar 21 20:11:42 2022 rev:31 rq:963482 version:1.5.21 Changes: -------- --- /work/SRC/openSUSE:Factory/borgmatic/borgmatic.changes 2021-09-16 23:17:27.523971420 +0200 +++ /work/SRC/openSUSE:Factory/.borgmatic.new.25692/borgmatic.changes 2022-03-21 20:11:56.468435198 +0100 @@ -1,0 +2,17 @@ +Wed Mar 16 06:36:36 UTC 2022 - Tuukka Pasanen <[email protected]> + +- Update 1.5.21 which is last that supports Python 3.6 + * #28: Optionally retry failing backups via "retries" and "retry_wait" configuration options. + * #306: Add "list_options" MySQL configuration option for passing additional arguments to MySQL + list command. + * #459: Add support for old version (2.x) of jsonschema library. + * #387: Fix error when configured source directories are not present on the filesystem at the time + of backup. Now, Borg will complain, but the backup will still continue. + * #455: Mention changing borgmatic path in cron documentation. + Update sample systemd service file with more granular read-only filesystem settings. + * Move Gitea and GitHub hosting from a personal namespace to an organization for better + collaboration with related projects. + * #389: Fix "message too long" error when logging to rsyslog. + * #440: Fix traceback that can occur when dumping a database. + +------------------------------------------------------------------- Old: ---- borgmatic-1.5.17.tar.gz New: ---- borgmatic-1.5.21.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ borgmatic.spec ++++++ --- /var/tmp/diff_new_pack.aFCY7w/_old 2022-03-21 20:11:57.012435741 +0100 +++ /var/tmp/diff_new_pack.aFCY7w/_new 2022-03-21 20:11:57.016435745 +0100 @@ -1,7 +1,7 @@ # # spec file for package borgmatic # -# Copyright (c) 2021 SUSE LLC +# Copyright (c) 2022 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed @@ -17,7 +17,7 @@ Name: borgmatic -Version: 1.5.17 +Version: 1.5.21 Release: 0 Summary: Automation tool for borgbackup License: GPL-3.0-only ++++++ borgmatic-1.5.17.tar.gz -> borgmatic-1.5.21.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/.drone.yml new/borgmatic-1.5.21/.drone.yml --- old/borgmatic-1.5.17/.drone.yml 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/.drone.yml 2021-11-22 22:19:15.000000000 +0100 @@ -14,6 +14,9 @@ MYSQL_ROOT_PASSWORD: test MYSQL_DATABASE: test +clone: + skip_verify: true + steps: - name: build image: alpine:3.9 @@ -36,6 +39,9 @@ MYSQL_ROOT_PASSWORD: test MYSQL_DATABASE: test +clone: + skip_verify: true + steps: - name: build image: alpine:3.10 @@ -58,6 +64,9 @@ MYSQL_ROOT_PASSWORD: test MYSQL_DATABASE: test +clone: + skip_verify: true + steps: - name: build image: alpine:3.13 @@ -68,11 +77,12 @@ kind: pipeline name: documentation +clone: + skip_verify: true + steps: - name: build - #image: plugins/docker - # Temporary work-around for https://github.com/drone-plugins/drone-docker/pull/327 - image: techknowlogick/drone-docker + image: plugins/docker settings: username: from_secret: docker_username @@ -82,5 +92,9 @@ dockerfile: docs/Dockerfile trigger: + repo: + - borgmatic-collective/borgmatic branch: - master + event: + - push diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/NEWS new/borgmatic-1.5.21/NEWS --- old/borgmatic-1.5.17/NEWS 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/NEWS 2021-11-22 22:19:15.000000000 +0100 @@ -1,3 +1,25 @@ +1.5.21 + * #28: Optionally retry failing backups via "retries" and "retry_wait" configuration options. + * #306: Add "list_options" MySQL configuration option for passing additional arguments to MySQL + list command. + * #459: Add support for old version (2.x) of jsonschema library. + +1.5.20 + * Re-release with correct version without dev0 tag. + +1.5.19 + * #387: Fix error when configured source directories are not present on the filesystem at the time + of backup. Now, Borg will complain, but the backup will still continue. + * #455: Mention changing borgmatic path in cron documentation. + * Update sample systemd service file with more granular read-only filesystem settings. + * Move Gitea and GitHub hosting from a personal namespace to an organization for better + collaboration with related projects. + * 1k ???s on GitHub! + +1.5.18 + * #389: Fix "message too long" error when logging to rsyslog. + * #440: Fix traceback that can occur when dumping a database. + 1.5.17 * #437: Fix error when configuration file contains "umask" option. * Remove test dependency on vim and /dev/urandom. @@ -564,7 +586,7 @@ * #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed includes/excludes. * Moved issue tracker from Taiga to integrated Gitea tracker at - https://projects.torsion.org/witten/borgmatic/issues + https://projects.torsion.org/borgmatic-collective/borgmatic/issues 1.1.12 * #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/README.md new/borgmatic-1.5.21/README.md --- old/borgmatic-1.5.17/README.md 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/README.md 2021-11-22 22:19:15.000000000 +0100 @@ -92,21 +92,20 @@ referral links, but without any tracking scripts or cookies.) <ul> - <li class="referral"><a href="https://www.rsync.net/cgi-bin/borg.cgi?campaign=borg&adgroup=borgmatic";>rsync.net</a>: Cloud Storage provider with full support for borg and any other SSH/SFTP tool</li> <li class="referral"><a href="https://www.borgbase.com/?utm_source=borgmatic";>BorgBase</a>: Borg hosting service with support for monitoring, 2FA, and append-only repos</li> <li class="referral"><a href="https://storage.lima-labs.com/special-pricing-offer-for-borgmatic-users/";>Lima-Labs</a>: Affordable, reliable cloud data storage accessable via SSH/SCP/FTP for Borg backups or any other bulk storage needs</li> </ul> -Additionally, [Hetzner](https://www.hetzner.com/storage/storage-box) has a -compatible storage offering, but does not currently fund borgmatic -development or hosting. +Additionally, [rsync.net](https://www.rsync.net/products/borg.html) and +[Hetzner](https://www.hetzner.com/storage/storage-box) have compatible storage +offerings, but do not currently fund borgmatic development or hosting. ## Support and contributing ### Issues You've got issues? Or an idea for a feature enhancement? We've got an [issue -tracker](https://projects.torsion.org/witten/borgmatic/issues). In order to +tracker](https://projects.torsion.org/borgmatic-collective/borgmatic/issues). In order to create a new issue or comment on an issue, you'll need to [login first](https://projects.torsion.org/user/login). Note that you can login with an existing GitHub account if you prefer. @@ -129,15 +128,15 @@ ### Contributing borgmatic [source code is -available](https://projects.torsion.org/witten/borgmatic) and is also mirrored -on [GitHub](https://github.com/witten/borgmatic) for convenience. +available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored +on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience. borgmatic is licensed under the GNU General Public License version 3 or any later version. If you'd like to contribute to borgmatic development, please feel free to -submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls) -or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first +submit a [Pull Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls) +or open an [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) first to discuss your idea. We also accept Pull Requests on GitHub, if that's more your thing. In general, contributions are very welcome. We don't bite! @@ -145,5 +144,5 @@ how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for info on cloning source code, running tests, etc. -<a href="https://build.torsion.org/witten/borgmatic"; alt="build status"></a> +<a href="https://build.torsion.org/borgmatic-collective/borgmatic"; alt="build status"></a> diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/borgmatic/borg/create.py new/borgmatic-1.5.21/borgmatic/borg/create.py --- old/borgmatic-1.5.17/borgmatic/borg/create.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/borgmatic/borg/create.py 2021-11-22 22:19:15.000000000 +0100 @@ -44,13 +44,18 @@ return tuple(os.path.expanduser(directory) for directory in directories) -def map_directories_to_devices(directories): # pragma: no cover +def map_directories_to_devices(directories): ''' Given a sequence of directories, return a map from directory to an identifier for the device on - which that directory resides. This is handy for determining whether two different directories - are on the same filesystem (have the same device identifier). + which that directory resides or None if the path doesn't exist. + + This is handy for determining whether two different directories are on the same filesystem (have + the same device identifier). ''' - return {directory: os.stat(directory).st_dev for directory in directories} + return { + directory: os.stat(directory).st_dev if os.path.exists(directory) else None + for directory in directories + } def deduplicate_directories(directory_devices): @@ -82,6 +87,7 @@ for parent in parents: if ( pathlib.PurePath(other_directory) == parent + and directory_devices[directory] is not None and directory_devices[other_directory] == directory_devices[directory] ): if directory in deduplicated: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/borgmatic/commands/borgmatic.py new/borgmatic-1.5.21/borgmatic/commands/borgmatic.py --- old/borgmatic-1.5.17/borgmatic/commands/borgmatic.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/borgmatic/commands/borgmatic.py 2021-11-22 22:19:15.000000000 +0100 @@ -4,6 +4,8 @@ import logging import os import sys +import time +from queue import Queue from subprocess import CalledProcessError import colorama @@ -52,6 +54,8 @@ local_path = location.get('local_path', 'borg') remote_path = location.get('remote_path') + retries = storage.get('retries', 0) + retry_wait = storage.get('retry_wait', 0) borg_environment.initialize(storage) encountered_error = None error_repository = '' @@ -120,7 +124,16 @@ ) if not encountered_error: - for repository_path in location['repositories']: + repo_queue = Queue() + for repo in location['repositories']: + repo_queue.put((repo, 0),) + + while not repo_queue.empty(): + repository_path, retry_num = repo_queue.get() + timeout = retry_num * retry_wait + if timeout: + logger.warning(f'{config_filename}: Sleeping {timeout}s before next retry') + time.sleep(timeout) try: yield from run_actions( arguments=arguments, @@ -134,11 +147,17 @@ repository_path=repository_path, ) except (OSError, CalledProcessError, ValueError) as error: - encountered_error = error - error_repository = repository_path yield from make_error_log_records( '{}: Error running actions for repository'.format(repository_path), error ) + if retry_num < retries: + repo_queue.put((repository_path, retry_num + 1),) + logger.warning( + f'{config_filename}: Retrying... attempt {retry_num + 1}/{retries}' + ) + continue + encountered_error = error + error_repository = repository_path if not encountered_error: try: @@ -257,7 +276,7 @@ hooks, local_path, remote_path, - repository_path + repository_path, ): # pragma: no cover ''' Given parsed command-line arguments as an argparse.ArgumentParser instance, several different diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/borgmatic/config/schema.yaml new/borgmatic-1.5.21/borgmatic/config/schema.yaml --- old/borgmatic-1.5.17/borgmatic/config/schema.yaml 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/borgmatic/config/schema.yaml 2021-11-22 22:19:15.000000000 +0100 @@ -135,12 +135,14 @@ type: string description: | Any paths matching these patterns are excluded from backups. - Globs and tildes are expanded. Do not backslash spaces in - path names. See the output of "borg help patterns" for more - details. + Globs and tildes are expanded. (Note however that a glob + pattern must either start with a glob or be an absolute + path.) Do not backslash spaces in path names. See the output + of "borg help patterns" for more details. example: - '*.pyc' - /home/*/.cache + - '*/.vim*.tmp' - /etc/ssl - /home/user/path with spaces exclude_from: @@ -249,6 +251,19 @@ Remote network upload rate limit in kiBytes/second. Defaults to unlimited. example: 100 + retries: + type: integer + description: | + Number of times to retry a failing backup before giving up. + Defaults to 0 (i.e., does not attempt retry). + example: 3 + retry_wait: + type: integer + description: | + Wait time between retries (in seconds) to allow transient + issues to pass. Increases after each retry as a form of + backoff. Defaults to 0 (no wait). + example: 10 temporary_directory: type: string description: | @@ -633,7 +648,7 @@ Password with which to connect to the database. Omitting a password will only work if PostgreSQL is configured to trust the configured username - without a password, or you create a ~/.pgpass + without a password or you create a ~/.pgpass file. example: trustsome1 format: @@ -734,6 +749,14 @@ configured to trust the configured username without a password. example: trustsome1 + list_options: + type: string + description: | + Additional mysql options to pass directly to + the mysql command that lists available + databases, without performing any validation on + them. See mysql documentation for details. + example: --defaults-extra-file=my.cnf options: type: string description: | diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/borgmatic/config/validate.py new/borgmatic-1.5.21/borgmatic/config/validate.py --- old/borgmatic-1.5.17/borgmatic/config/validate.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/borgmatic/config/validate.py 2021-11-22 22:19:15.000000000 +0100 @@ -110,7 +110,10 @@ override.apply_overrides(config, overrides) normalize.normalize(config) - validator = jsonschema.Draft7Validator(schema) + try: + validator = jsonschema.Draft7Validator(schema) + except AttributeError: # pragma: no cover + validator = jsonschema.Draft4Validator(schema) validation_errors = tuple(validator.iter_errors(config)) if validation_errors: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/borgmatic/execute.py new/borgmatic-1.5.21/borgmatic/execute.py --- old/borgmatic-1.5.17/borgmatic/execute.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/borgmatic/execute.py 2021-11-22 22:19:15.000000000 +0100 @@ -81,6 +81,7 @@ for other_process in processes: if ( other_process.poll() is None + and other_process.stdout and other_process.stdout not in output_buffers ): # Add the process's output to output_buffers to ensure it'll get read. @@ -138,9 +139,12 @@ if not output_buffer: continue - remaining_output = output_buffer.read().rstrip().decode() + while True: # pragma: no cover + remaining_output = output_buffer.readline().rstrip().decode() + + if not remaining_output: + break - if remaining_output: # pragma: no cover logger.log(output_log_level, remaining_output) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/borgmatic/hooks/mysql.py new/borgmatic-1.5.21/borgmatic/hooks/mysql.py --- old/borgmatic-1.5.17/borgmatic/hooks/mysql.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/borgmatic/hooks/mysql.py 2021-11-22 22:19:15.000000000 +0100 @@ -31,6 +31,7 @@ show_command = ( ('mysql',) + + (tuple(database['list_options'].split(' ')) if 'list_options' in database else ()) + (('--host', database['hostname']) if 'hostname' in database else ()) + (('--port', str(database['port'])) if 'port' in database else ()) + (('--protocol', 'tcp') if 'hostname' in database or 'port' in database else ()) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/docs/_includes/components/suggestion-link.html new/borgmatic-1.5.21/docs/_includes/components/suggestion-link.html --- old/borgmatic-1.5.17/docs/_includes/components/suggestion-link.html 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/docs/_includes/components/suggestion-link.html 2021-11-22 22:19:15.000000000 +0100 @@ -1,17 +1,5 @@ <h2>Improve this documentation</h2> <p>Have an idea on how to make this documentation even better? Use our <a -href="https://projects.torsion.org/witten/borgmatic/issues";>issue tracker</a> to send your +href="https://projects.torsion.org/borgmatic-collective/borgmatic/issues";>issue tracker</a> to send your feedback!</p> - -<script> - document.getElementById('_page').value = window.location.href; - window.sk=window.sk||function(){(sk.q=sk.q||[]).push(arguments)}; - - sk('form', 'init', { - id: '1d536680ab96', - element: '#suggestion-form' - }); -</script> - -<script defer src="https://js.statickit.com/statickit.js";></script> diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/docs/how-to/develop-on-borgmatic.md new/borgmatic-1.5.21/docs/how-to/develop-on-borgmatic.md --- old/borgmatic-1.5.17/docs/how-to/develop-on-borgmatic.md 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/docs/how-to/develop-on-borgmatic.md 2021-11-22 22:19:15.000000000 +0100 @@ -10,17 +10,17 @@ To get set up to hack on borgmatic, first clone master via HTTPS or SSH: ```bash -git clone https://projects.torsion.org/witten/borgmatic.git +git clone https://projects.torsion.org/borgmatic-collective/borgmatic.git ``` Or: ```bash -git clone ssh://[email protected]:3022/witten/borgmatic.git +git clone ssh://[email protected]:3022/borgmatic-collective/borgmatic.git ``` Then, install borgmatic -"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)" +"[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs)" so that you can run borgmatic commands while you're hacking on them to make sure your changes work. @@ -66,8 +66,6 @@ tox -e black ``` -Note that Black requires at minimum Python 3.6. - And if you get a complaint from the [isort](https://github.com/timothycrosley/isort) Python import orderer, you can ask isort to order your imports for you: @@ -118,7 +116,7 @@ Each pull request triggers a continuous integration build which runs the test suite. You can view these builds on -[build.torsion.org](https://build.torsion.org/witten/borgmatic), and they're +[build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic), and they're also linked from the commits list on each pull request. ## Documentation development diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/docs/how-to/set-up-backups.md new/borgmatic-1.5.21/docs/how-to/set-up-backups.md --- old/borgmatic-1.5.17/docs/how-to/set-up-backups.md 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/docs/how-to/set-up-backups.md 2021-11-22 22:19:15.000000000 +0100 @@ -28,7 +28,7 @@ This installs borgmatic and its commands at the `/root/.local/bin` path. Your pip binary may have a different name than "pip3". Make sure you're using -Python 3, as borgmatic does not support Python 2. +Python 3.6+, as borgmatic does not support Python 2. The next step is to ensure that borgmatic's commands available are on your system `PATH`, so that you can run borgmatic: @@ -77,7 +77,7 @@ Besides the approaches described above, there are several other options for installing borgmatic: - * [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) + * [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) (+ Docker Compose files) * [Docker base image](https://hub.docker.com/r/monachus/borgmatic/) * [Debian](https://tracker.debian.org/pkg/borgmatic) * [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic) @@ -100,14 +100,13 @@ referral links, but without any tracking scripts or cookies.) <ul> - <li class="referral"><a href="https://www.rsync.net/cgi-bin/borg.cgi?campaign=borg&adgroup=borgmatic";>rsync.net</a>: Cloud Storage provider with full support for borg and any other SSH/SFTP tool</li> <li class="referral"><a href="https://www.borgbase.com/?utm_source=borgmatic";>BorgBase</a>: Borg hosting service with support for monitoring, 2FA, and append-only repos</li> <li class="referral"><a href="https://storage.lima-labs.com/special-pricing-offer-for-borgmatic-users/";>Lima-Labs</a>: Affordable, reliable cloud data storage accessable via SSH/SCP/FTP for Borg backups or any other bulk storage needs</li> </ul> -Additionally, [Hetzner](https://www.hetzner.com/storage/storage-box) has a -compatible storage offering, but does not currently fund borgmatic -development or hosting. +Additionally, [rsync.net](https://www.rsync.net/products/borg.html) and +[Hetzner](https://www.hetzner.com/storage/storage-box) have compatible storage +offerings, but do not currently fund borgmatic development or hosting. ## Configuration @@ -250,7 +249,7 @@ ### cron If you're using cron, download the [sample cron -file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic). +file](https://projects.torsion.org/borgmatic-collective/borgmatic/src/master/sample/cron/borgmatic). Then, from the directory where you downloaded it: ```bash @@ -258,7 +257,10 @@ sudo chmod +x /etc/cron.d/borgmatic ``` -You can modify the cron file if you'd like to run borgmatic more or less frequently. +If borgmatic is installed at a different location than +`/root/.local/bin/borgmatic`, edit the cron file with the correct path. You +can also modify the cron file if you'd like to run borgmatic more or less +frequently. ### systemd @@ -271,9 +273,9 @@ be able to skip some of the steps below.) First, download the [sample systemd service -file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service) +file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.service) and the [sample systemd timer -file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer). +file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer). Then, from the directory where you downloaded them: @@ -294,7 +296,7 @@ If you run borgmatic in macOS with launchd, you may encounter permissions issues when reading files to backup. If that happens to you, you may be interested in an [unofficial work-around for Full Disk -Access](https://projects.torsion.org/witten/borgmatic/issues/293). +Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293). ## Colored output diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/sample/systemd/borgmatic.service new/borgmatic-1.5.21/sample/systemd/borgmatic.service --- old/borgmatic-1.5.17/sample/systemd/borgmatic.service 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/sample/systemd/borgmatic.service 2021-11-22 22:19:15.000000000 +0100 @@ -32,13 +32,16 @@ SystemCallArchitectures=native SystemCallFilter=@system-service SystemCallErrorNumber=EPERM -# Restrict write access -# Change to 'ProtectSystem=strict' and uncomment 'ProtectHome' to make the whole file -# system read-only be default and uncomment 'ReadWritePaths' for the required write access. -# Add local repositroy paths to the list of 'ReadWritePaths' like '-/mnt/my_backup_drive'. +# To restrict write access further, change "ProtectSystem" to "strict" and uncomment +# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository +# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This +# leaves most of the filesystem read-only to borgmatic. ProtectSystem=full -# ProtectHome=read-only -# ReadWritePaths=-/root/.config/borg -/root/.cache/borg -/root/.borgmatic +# ReadWritePaths=-/mnt/my_backup_drive +# ReadOnlyPaths=-/var/lib/my_backup_source +# This will mount a tmpfs on top of /root and pass through needed paths +# ProtectHome=tmpfs +# BindPaths=-/root/.cache/borg -/root/.cache/borg -/root/.borgmatic CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/scripts/release new/borgmatic-1.5.21/scripts/release --- old/borgmatic-1.5.17/scripts/release 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/scripts/release 2021-11-22 22:19:15.000000000 +0100 @@ -38,7 +38,7 @@ release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')" escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')" curl --silent --request POST \ - "https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases"; \ + "https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases"; \ --header "Authorization: token $projects_token" \ --header "Accept: application/json" \ --header "Content-Type: application/json" \ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/scripts/run-full-tests new/borgmatic-1.5.21/scripts/run-full-tests --- old/borgmatic-1.5.17/scripts/run-full-tests 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/scripts/run-full-tests 2021-11-22 22:19:15.000000000 +0100 @@ -13,8 +13,8 @@ apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client # If certain dependencies of black are available in this version of Alpine, install them. apk add --no-cache py3-typed-ast py3-regex || true -python3 -m pip install --upgrade pip==20.2.4 setuptools==50.3.2 -pip3 install tox==3.20.1 +python3 -m pip install --upgrade pip==21.3.1 setuptools==58.2.0 +pip3 install tox==3.24.4 export COVERAGE_FILE=/tmp/.coverage tox --workdir /tmp/.tox --sitepackages tox --workdir /tmp/.tox --sitepackages -e end-to-end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/setup.py new/borgmatic-1.5.21/setup.py --- old/borgmatic-1.5.17/setup.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/setup.py 2021-11-22 22:19:15.000000000 +0100 @@ -1,6 +1,6 @@ from setuptools import find_packages, setup -VERSION = '1.5.17' +VERSION = '1.5.21' setup( diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/tests/integration/test_execute.py new/borgmatic-1.5.21/tests/integration/test_execute.py --- old/borgmatic-1.5.17/tests/integration/test_execute.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/tests/integration/test_execute.py 2021-11-22 22:19:15.000000000 +0100 @@ -162,6 +162,37 @@ ) +def test_log_outputs_does_not_error_when_one_process_exits(): + flexmock(module.logger).should_receive('log') + flexmock(module).should_receive('command_for_process').and_return('grep') + + process = subprocess.Popen( + [ + sys.executable, + '-c', + "import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))", + ], + stdout=None, # Specifically test the case of a process without stdout captured. + stderr=None, + ) + other_process = subprocess.Popen( + ['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + flexmock(module).should_receive('output_buffer_for_process').with_args( + process, (process.stdout,) + ).and_return(process.stderr) + flexmock(module).should_receive('output_buffer_for_process').with_args( + other_process, (process.stdout,) + ).and_return(other_process.stdout) + + module.log_outputs( + (process, other_process), + exclude_stdouts=(process.stdout,), + output_log_level=logging.INFO, + borg_local_path='borg', + ) + + def test_log_outputs_truncates_long_error_output(): flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0 flexmock(module.logger).should_receive('log') diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/tests/unit/borg/test_create.py new/borgmatic-1.5.21/tests/unit/borg/test_create.py --- old/borgmatic-1.5.17/tests/unit/borg/test_create.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/tests/unit/borg/test_create.py 2021-11-22 22:19:15.000000000 +0100 @@ -60,6 +60,30 @@ assert paths == () +def test_map_directories_to_devices_gives_device_id_per_path(): + flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55)) + flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66)) + + device_map = module.map_directories_to_devices(('/foo', '/bar')) + + assert device_map == { + '/foo': 55, + '/bar': 66, + } + + +def test_map_directories_to_devices_with_missing_path_does_not_error(): + flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55)) + flexmock(module.os).should_receive('stat').with_args('/bar').and_raise(FileNotFoundError) + + device_map = module.map_directories_to_devices(('/foo', '/bar')) + + assert device_map == { + '/foo': 55, + '/bar': None, + } + + @pytest.mark.parametrize( 'directories,expected_directories', ( @@ -72,6 +96,7 @@ ({'/root': 1, '/root/foo/': 1}, ('/root',)), ({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')), ({'/root/foo': 1, '/root': 1}, ('/root',)), + ({'/root': None, '/root/foo': None}, ('/root', '/root/foo')), ({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')), ({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)), ({'/dup': 1, '/dup': 1}, ('/dup',)), diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/tests/unit/commands/test_borgmatic.py new/borgmatic-1.5.21/tests/unit/commands/test_borgmatic.py --- old/borgmatic-1.5.17/tests/unit/commands/test_borgmatic.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/tests/unit/commands/test_borgmatic.py 2021-11-22 22:19:15.000000000 +0100 @@ -1,5 +1,6 @@ import logging import subprocess +import time from flexmock import flexmock @@ -184,6 +185,160 @@ assert results == expected_results +def test_run_configuration_retries_soft_error(): + # Run action first fails, second passes + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).and_return([]) + expected_results = [flexmock()] + flexmock(module).should_receive('make_error_log_records').and_return(expected_results).once() + config = {'location': {'repositories': ['foo']}, 'storage': {'retries': 1}} + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + +def test_run_configuration_retries_hard_error(): + # Run action fails twice + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).times(2) + expected_results = [flexmock(), flexmock()] + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[:1]).with_args( + 'foo: Error running actions for repository', OSError + ).and_return( + expected_results[1:] + ).twice() + config = {'location': {'repositories': ['foo']}, 'storage': {'retries': 1}} + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + +def test_run_repos_ordered(): + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).times(2) + expected_results = [flexmock(), flexmock()] + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[:1]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[1:]).ordered() + config = {'location': {'repositories': ['foo', 'bar']}} + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + +def test_run_configuration_retries_round_robbin(): + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).times(4) + expected_results = [flexmock(), flexmock(), flexmock(), flexmock()] + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[0:1]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[1:2]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[2:3]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[3:4]).ordered() + config = {'location': {'repositories': ['foo', 'bar']}, 'storage': {'retries': 1}} + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + +def test_run_configuration_retries_one_passes(): + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).and_raise(OSError).and_return( + [] + ).and_raise(OSError).times(4) + expected_results = [flexmock(), flexmock(), flexmock()] + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[0:1]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[1:2]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[2:3]).ordered() + config = {'location': {'repositories': ['foo', 'bar']}, 'storage': {'retries': 1}} + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + +def test_run_configuration_retry_wait(): + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).times(4) + expected_results = [flexmock(), flexmock(), flexmock(), flexmock()] + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[0:1]).ordered() + + flexmock(time).should_receive('sleep').with_args(10).and_return().ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[1:2]).ordered() + + flexmock(time).should_receive('sleep').with_args(20).and_return().ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[2:3]).ordered() + + flexmock(time).should_receive('sleep').with_args(30).and_return().ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[3:4]).ordered() + config = {'location': {'repositories': ['foo']}, 'storage': {'retries': 3, 'retry_wait': 10}} + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + +def test_run_configuration_retries_timeout_multiple_repos(): + flexmock(module.borg_environment).should_receive('initialize') + flexmock(module.command).should_receive('execute_hook') + flexmock(module).should_receive('run_actions').and_raise(OSError).and_raise(OSError).and_return( + [] + ).and_raise(OSError).times(4) + expected_results = [flexmock(), flexmock(), flexmock()] + flexmock(module).should_receive('make_error_log_records').with_args( + 'foo: Error running actions for repository', OSError + ).and_return(expected_results[0:1]).ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[1:2]).ordered() + + # Sleep before retrying foo (and passing) + flexmock(time).should_receive('sleep').with_args(10).and_return().ordered() + + # Sleep before retrying bar (and failing) + flexmock(time).should_receive('sleep').with_args(10).and_return().ordered() + flexmock(module).should_receive('make_error_log_records').with_args( + 'bar: Error running actions for repository', OSError + ).and_return(expected_results[2:3]).ordered() + config = { + 'location': {'repositories': ['foo', 'bar']}, + 'storage': {'retries': 1, 'retry_wait': 10}, + } + arguments = {'global': flexmock(monitoring_verbosity=1, dry_run=False), 'create': flexmock()} + results = list(module.run_configuration('test.yaml', config, arguments)) + assert results == expected_results + + def test_load_configurations_collects_parsed_configurations(): configuration = flexmock() other_configuration = flexmock() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/borgmatic-1.5.17/tests/unit/hooks/test_mysql.py new/borgmatic-1.5.21/tests/unit/hooks/test_mysql.py --- old/borgmatic-1.5.17/tests/unit/hooks/test_mysql.py 2021-07-27 19:04:22.000000000 +0200 +++ new/borgmatic-1.5.21/tests/unit/hooks/test_mysql.py 2021-11-22 22:19:15.000000000 +0100 @@ -198,6 +198,24 @@ assert module.dump_databases(databases, 'test.yaml', {}, dry_run=False) == [process] +def test_database_names_to_dump_runs_mysql_with_list_options(): + database = {'name': 'all', 'list_options': '--defaults-extra-file=my.cnf'} + flexmock(module).should_receive('execute_command').with_args( + ( + 'mysql', + '--defaults-extra-file=my.cnf', + '--skip-column-names', + '--batch', + '--execute', + 'show schemas', + ), + output_log_level=None, + extra_environment=None, + ).and_return(('foo\nbar')).once() + + assert module.database_names_to_dump(database, None, 'test.yaml', '') == ('foo', 'bar') + + def test_dump_databases_errors_for_missing_all_databases(): databases = [{'name': 'all'}] process = flexmock()
