Tim,
FYI: we publish everything to a new, empty directory. When this is done
we change the symlink that points to the 'current' directory, to this
new dir. Rsync follows the symlink. I believe this is similar to other
implementations.
This does not prevent mid-transfer issues as such, but.. the change
itself is fairly atomic. In other words if the RP suspects a
mid-transfer update may have occurred, and then restarts the transfer:
they should get a consistent view.
Since the manifest is always updated when changes occur, and a new EE
certificate issued with not-before time set to the moment of
publication, the RP can re-fetch the manifest and check whether it was
updated.
So, when processing the SIA for a cert:
1) get manifest (follow manifest pointer)
2) get all other objects (follow dir pointer)
3) get manifest again and compare to 1
--> if 3 and 1 don't match, wait a moment and repeat.
I am not sure though whether this enough.. other ideas?
Cheers
Tim
I like your overall approach, although not the details of the algorithm :-).
I prefer a description that does not assume that the RP software is
always extracting SIA URLs to drive the rsync operation in realtime.
For example, our RP software builds a list of URLs from validated
certs (as well as from trust anchors) and we visit directories based
on that list. We update the list anachronously, based on later stage
processing of certs. So, if you believe the following procedure is
equivalent, wrt detecting and responding to fetching files from a
directory that is being updated, I'd be comfortable:
1- retrieve changed files from a directory (what rsync does by default)
2- wait a moment (how long?) and ask rsync to do this again, for the
same directory
3- if no changes are detected, hence no files returned, then assume that
the files that were retrieved in step 1 are a consistent; reset the counter
for this directory to 0.
OTHERWISE increment the counter for this directory; if the counter
is greater
than a threshold value, give up on this directory for now, and
note this error,
OTHERWISE wait a moment and return to step 1.
This description does not require any "deep" processing of the
retrieved objects,
it gives the directory time to stabilize, and allows for a max number
of retries, without getting stuck. It does not rely on the manifest
being updated in a particular order. It does have the downside of
making rsync to visit every directory at least twice. I'm not enough
of an rsync guy to know if Chris's command line parameters are a
simple way to implement the semantics of my step 2 above. Chris?
Steve
_______________________________________________
sidr mailing list
sidr@ietf.org
https://www.ietf.org/mailman/listinfo/sidr
