Peter Palfrader pushed to branch master at snapshot / snapshot
Commits:
e7780d89 by Peter Palfrader at 2025-04-12T12:42:12+02:00
sync-farm-incremental: run 4 syncers in parallel
- - - - -
e080b7b3 by Peter Palfrader at 2025-04-12T13:03:22+02:00
Add sync-farm-incremental ssh trigger that uses a systemd service (so we have
sane logging and error reporting)
- - - - -
3 changed files:
- + mirror/snapshot-sync-farm-incremental.service.sample
- mirror/sync-farm-incremental
- + mirror/sync-farm-incremental-ssh-trigger
Changes:
=====================================
mirror/snapshot-sync-farm-incremental.service.sample
=====================================
@@ -0,0 +1,5 @@
+[Unit]
+Description=snapshot incremental farm sync
+
+[Service]
+ExecStart=/home/snapshot/code/mirror/sync-farm-incremental
=====================================
mirror/sync-farm-incremental
=====================================
@@ -3,20 +3,43 @@
set -e
set -o pipefail
-farm="$(readlink -f farm)"
-cd "$farm"
-for i in *; do
- exec 200< "$i"
+if [ "$#" = 2 ]; then
+ farm="$1"; shift
+ item="$1"; shift
+ echo >&2 "Doing ${item}"
+
+ cd "$farm"
+
+ exec 200< "$item"
flock -e 200
- tmpdir="$(mktemp -d --tmpdir="$i" ".incremental-update-XXXXXX")"
+ tmpdir="$(mktemp -d --tmpdir="$item" ".incremental-update-XXXXXX")"
tmpdir="$(readlink -f "$tmpdir")"
trap "rm -rf '$tmpdir'" EXIT
- ~/code/mirror/farm-list-entries --farmpath "$farm" "$i" |
- ssh -4 -C snapshot-mlm-01.debian.org farm-make-update-tarball "$i" |
+ ~/code/mirror/farm-list-entries --farmpath "$farm" "$item" |
+ ssh -4 -C snapshot-mlm-01.debian.org farm-make-update-tarball "$item" |
(cd $tmpdir && tar xaf -)
~/code/mirror/process-farm-update --farmpath "$farm" "$tmpdir"
rm -rf "$tmpdir"
-done
+elif [ "$#" = 0 ]; then
+ self="${BASH_SOURCE[0]}"
+
+ farm="$(readlink -f farm)"
+ while :; do
+ rm -vf "${farm}/.incremental-sync-needed"
+
+ find "$farm" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | \
+ grep '^[0-9a-f][0-9a-f]$' | \
+ sort -R | \
+ xargs -n 1 -P 4 "$self" "$farm"
+
+ [ -e "${farm}/.incremental-sync-needed" ] || break
+ echo "Re-running because ${farm}/.incremental-sync-needed exists again."
+ done
+else
+ echo >&2 "Usage: $0"
+ echo >&2 "Usage: $0 <farmpath> <dir-to-process [0-9a-f][0-9a-f]>"
+ exit 1
+fi
=====================================
mirror/sync-farm-incremental-ssh-trigger
=====================================
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+set -e
+set -o pipefail
+farm="$(readlink -f farm)"
+
+touch "${farm}/.incremental-sync-needed"
+systemctl --user start snapshot-sync-farm-incremental.service
View it on GitLab:
https://salsa.debian.org/snapshot-team/snapshot/-/compare/ee743e3cf2cd64575b3ee05f44994d13f10761bb...e080b7b39cc706f0b5d92a4543cb28115791356e
--
View it on GitLab:
https://salsa.debian.org/snapshot-team/snapshot/-/compare/ee743e3cf2cd64575b3ee05f44994d13f10761bb...e080b7b39cc706f0b5d92a4543cb28115791356e
You're receiving this email because of your account on salsa.debian.org.
