Your message dated Sat, 01 Apr 2023 18:40:48 +0000
with message-id <[email protected]>
and subject line Bug#1033627: fixed in git-annex 10.20230126-3
has caused the Debian Bug report #1033627,
regarding exporttree=yes data loss
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1033627: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033627
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: git-annex
Version: 10.20230126-2
Severity: important
git-annex has this bug, which can result in data loss when using an
external special remote with exporttree=yes
https://git-annex.branchable.com/bugs/external_remote_export_sent_to_wrong_process/
This is fixed in version 10.20230329.
I've attached a patch series that fixes this. Note that the second patch
is not strictly necessary, but it's all that's needed to support
VERSION 2, which some external special remote programs will now be
using to avoid being used with the buggy git-annex. So, I strongly
recommend including it.
(The patches are lightly trimmed versions of upstream commits
02662f52920e84cd9464641ada84f6c3bbe3f86a and
18d326cb6f27912542f41fbb9525eefdbd553d09)
--
see shy jo
From 02662f52920e84cd9464641ada84f6c3bbe3f86a Mon Sep 17 00:00:00 2001
From: Joey Hess <[email protected]>
Date: Tue, 28 Mar 2023 15:21:10 -0400
Subject: [PATCH 1/6] fix concurrency bug causing EXPORT to be sent to the
wrong external
Fix bug that caused broken protocol to be used with external remotes that
use exporttree=yes. In some cases this could result in the wrong content
being exported to, or retrieved from the remote.
Sponsored-by: Nicholas Golder-Manning on Patreon
diff --git a/Remote/External.hs b/Remote/External.hs
index 4077030fa3..352b9eb68d 100644
--- a/Remote/External.hs
+++ b/Remote/External.hs
@@ -377,19 +377,27 @@ handleRequest external req mp responsehandler =
handleRequest' st external req mp responsehandler
handleRequestKey :: External -> (SafeKey -> Request) -> Key -> Maybe MeterUpdate -> ResponseHandler a -> Annex a
-handleRequestKey external mkreq k mp responsehandler = case mkSafeKey k of
- Right sk -> handleRequest external (mkreq sk) mp responsehandler
+handleRequestKey external mkreq k mp responsehandler =
+ withSafeKey k $ \sk -> handleRequest external (mkreq sk) mp responsehandler
+
+withSafeKey :: Key -> (SafeKey -> Annex a) -> Annex a
+withSafeKey k a = case mkSafeKey k of
+ Right sk -> a sk
Left e -> giveup e
{- Export location is first sent in an EXPORT message before
- the main request. This is done because the ExportLocation can
- contain spaces etc. -}
handleRequestExport :: External -> ExportLocation -> (SafeKey -> Request) -> Key -> Maybe MeterUpdate -> ResponseHandler a -> Annex a
-handleRequestExport external loc mkreq k mp responsehandler = do
- withExternalState external $ \st -> do
- checkPrepared st external
- sendMessage st (EXPORT loc)
- handleRequestKey external mkreq k mp responsehandler
+handleRequestExport external loc mkreq k mp responsehandler =
+ withSafeKey k $ \sk ->
+ -- Both the EXPORT and subsequent request must be sent to the
+ -- same external process, so run both with the same external
+ -- state.
+ withExternalState external $ \st -> do
+ checkPrepared st external
+ sendMessage st (EXPORT loc)
+ handleRequest' st external (mkreq sk) mp responsehandler
handleRequest' :: ExternalState -> External -> Request -> Maybe MeterUpdate -> ResponseHandler a -> Annex a
handleRequest' st external req mp responsehandler
--
2.39.1
From 18d326cb6f27912542f41fbb9525eefdbd553d09 Mon Sep 17 00:00:00 2001
From: Joey Hess <[email protected]>
Date: Tue, 28 Mar 2023 17:00:08 -0400
Subject: [PATCH 4/6] external protocol VERSION 2
Support VERSION 2 in the external special remote protocol, which is
identical to VERSION 1, but avoids external remote programs neededing to
work around the above bug. External remote program that support
exporttree=yes are recommended to be updated to send VERSION 2.
Sponsored-by: Kevin Mueller on Patreon
diff --git a/Remote/External/Types.hs b/Remote/External/Types.hs
index 894f09dc30..633dc641bd 100644
--- a/Remote/External/Types.hs
+++ b/Remote/External/Types.hs
@@ -415,7 +415,7 @@ newtype JobId = JobId Integer
deriving (Eq, Ord, Show)
supportedProtocolVersions :: [ProtocolVersion]
-supportedProtocolVersions = [1]
+supportedProtocolVersions = [1, 2]
instance Proto.Serializable JobId where
serialize (JobId n) = show n
diff --git a/doc/design/external_special_remote_protocol.mdwn b/doc/design/external_special_remote_protocol.mdwn
index 3d2f0588ae..665687f5be 100644
--- a/doc/design/external_special_remote_protocol.mdwn
+++ b/doc/design/external_special_remote_protocol.mdwn
@@ -39,7 +39,7 @@ empty, but the separating spaces are still required in that case.
The special remote is responsible for sending the first message, indicating
the version of the protocol it is using.
- VERSION 1
+ VERSION 2
Recent versions of git-annex respond with a message indicating
protocol extensions that it supports. Older versions of
@@ -271,7 +271,7 @@ These messages may be sent by the special remote at any time that it's
handling a request.
* `VERSION Int`
- Supported protocol version. Current version is 1. Must be sent first
+ Supported protocol version. Current version is 2. Must be sent first
thing at startup, as until it sees this git-annex does not know how to
talk with the special remote program!
(git-annex does not send a reply to this message, but may give up if it
@@ -428,6 +428,18 @@ remote.
git-annex will not talk to it any further. If the program receives
an ERROR from git-annex, it can exit with its own ERROR.
+## protocol versions
+
+Currently git-annex supports `VERSION 1` and `VERSION 2`.
+The two protocol versions are actually identical.
+
+Old versions of git-annex that supported only `VERSION 1`
+had a bug in their implementation of the
+part of the protocol documented in the [[export_and_import_appendix]].
+The bug could result in ontent being exported to the wrong file.
+External special remotes that implement that should use `VERSION 2` to
+avoid talking to the buggy old version of git-annex.
+
## extensions
These protocol extensions are currently supported.
diff --git a/doc/special_remotes/external/example.sh b/doc/special_remotes/external/example.sh
index cfae74ae25..413065d1e0 100755
--- a/doc/special_remotes/external/example.sh
+++ b/doc/special_remotes/external/example.sh
@@ -150,7 +150,7 @@ doremove () {
}
# This has to come first, to get the protocol started.
-echo VERSION 1
+echo VERSION 2
while read line; do
set -- $line
diff --git a/doc/special_remotes/external/git-annex-remote-ipfs b/doc/special_remotes/external/git-annex-remote-ipfs
index f255f9bc59..f546a1ae81 100755
--- a/doc/special_remotes/external/git-annex-remote-ipfs
+++ b/doc/special_remotes/external/git-annex-remote-ipfs
@@ -54,7 +54,7 @@ getaddrs () {
}
# This has to come first, to get the protocol started.
-echo VERSION 1
+echo VERSION 2
while read line; do
set -- $line
diff --git a/doc/special_remotes/external/git-annex-remote-torrent b/doc/special_remotes/external/git-annex-remote-torrent
index a3b2bb8580..f41585b7c2 100755
--- a/doc/special_remotes/external/git-annex-remote-torrent
+++ b/doc/special_remotes/external/git-annex-remote-torrent
@@ -100,7 +100,7 @@ downloadtorrent () {
}
# This has to come first, to get the protocol started.
-echo VERSION 1
+echo VERSION 2
while read line; do
set -- $line
--
2.39.1
signature.asc
Description: PGP signature
--- End Message ---
--- Begin Message ---
Source: git-annex
Source-Version: 10.20230126-3
Done: Sean Whitton <[email protected]>
We believe that the bug you reported is fixed in the latest version of
git-annex, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to [email protected],
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Sean Whitton <[email protected]> (supplier of updated git-annex package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [email protected])
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Sat, 01 Apr 2023 10:57:41 -0700
Source: git-annex
Architecture: source
Version: 10.20230126-3
Distribution: unstable
Urgency: medium
Maintainer: Debian Haskell Group
<[email protected]>
Changed-By: Sean Whitton <[email protected]>
Closes: 1033627
Changes:
git-annex (10.20230126-3) unstable; urgency=medium
.
* Backport fixes for data loss bug (Closes: #1033627).
Thanks to Joey Hess for the report and patches.
Checksums-Sha1:
6431860fd9777ec8e46a9153891751a5f9fdd60a 5558 git-annex_10.20230126-3.dsc
fe121d572b8bf5329299091919667dd7855188f1 80760
git-annex_10.20230126-3.debian.tar.xz
Checksums-Sha256:
809fef1f49805ca35f6ba36b15ac79e13c203fa152dad6a3f56ec48720946464 5558
git-annex_10.20230126-3.dsc
d7c0962fcb20c0fed3a5954240c65740c9f14e909f1ad54878a62d3cd7140534 80760
git-annex_10.20230126-3.debian.tar.xz
Files:
0a347cbffc7d8e82440f8da0075ff8f0 5558 utils optional
git-annex_10.20230126-3.dsc
fed5f7c470745fed79d1c16d8edda57c 80760 utils optional
git-annex_10.20230126-3.debian.tar.xz
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmQodDwACgkQaVt65L8G
YkAHVA//TjZD+lTsy4Ypk5kxe7Qpj8c4T2WapXbDaKPEUXOtNsSRDoBDn3/Dz30d
X95KuRot1SR1fBDbMrGoajSSVwk++yC0H+tgvw7Xa/0N6rzPEhOzisr2uxWLFrsc
5XQAGNCJLEzhuWgl00VEsYcxvesqxH+WL3tgEFZDm1mKFphKq4YNzq7mLNeh42WX
/IJjPBFCgWFAmDVEBqtosC8LOsr90U/uDaE6g0pUW9nQD2f91SDD/m9D//unljBN
uPezh/tkZp8WfwXLFANCMKTopkmjoFl51rmuvIA1aBAdq8RBC18VMl2mlfa9YKwg
RJ8jK20/MRwLKOhEHYce8+b2Ab0LMqpxhpfdyZCcz/vD1WIgk7CganzWaK+rgSbc
WWYTPuY3N4cYcUVby+1u+GnomGPvI1wIz7jbcLai31TfMQLo4TKOzZCboqtQilrX
7zYBcTP0yUEN9YtbxkrPFTU8IM0Cr7C7rDaDihdsos6KX+hU/OKjcuKOBRwhkIkd
Al+Y0O+IsfYgrOfKDuB6EFSJ9PMtJdlFZ9wkyNJe772hTIuQfwacrnDqW//nLU1d
ixZECYNh8gP66t84uU7HljJf6ceHuaNGBtDBSgQSLAXFKItfzp/BsJeo8242BWve
Q/4+vN9OGOVwqD4z3USwtDrvMCxDm59pebiBCxaRSYH45lE2rSc=
=guXb
-----END PGP SIGNATURE-----
--- End Message ---
