Is this being accessed via a fuse mount? If so it seems like that is
where this kind of feature should be implemented (like a mount option to
decide how to handle such files). Rsync shouldn't need special features
to deal with every kind of file storage.
On 9/12/23 05:22, Brian "bex" Exelbierd via rsync wrote:
Hi,
I have also posted this on GitHub but it isn’t clear that was the right
place: https://github.com/WayneD/rsync/issues/522
iCloud Drive will evict files that are unused or when additional space
is needed on the local drive. The evicted files are replace by
"bookmark" files that allow MacOS to continue to report the files in the
file system as though they were actually present. The files are
downloaded again either on request or when needed.
rsync, like all similar tools I can find, doesn't have any way of
handling these evicted files. I have been thinking about this and I
think I know how to make it work.
The short explanation is that iCloud Drive preserves access to the
required metadata. Here is an abbreviated output from `mdls` of an
evicted file:
```
% mdls .1998-tax-return.pdf.icloud
kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000
kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000
kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000
kMDItemDateAdded = 2021-11-05 15:42:15 +0000
kMDItemDisplayName = "1998-tax-return.pdf"
kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000
kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000
kMDItemFSSize = 1929932
kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000
kMDItemLogicalSize = 1929932
kMDItemPhysicalSize = 1929932
```
This metadata, I think, is enough to pass the rsync quick check as
described in the man page. Therefore, I suspect what needs to happen,
from a code perspective, is that rsync needs to be modified to do the
following when it finds an evicted file. All evicted files are named
consistently, `.<filename>.iCloud` so they are easy to spot.
1. Perform the system call equivalent of the mdls above to obtain the
appropriate dates and sizes.
2. If the file cannot be skipped because it either has changed or a
checksum is required, perform the system call equivalent of `brctl
download <filename>` to get the file downloaded.
3. Rsync the file
4. If the file was downloaded by rsync, perform the system call
equivalent of `brctl evict <filename>` to remove the file to leave the
system in the same state.
This simplistic algorithm would leave some open issues/caveats:
- It is likely that the rsync can only be run one-way from iCloud Drive
to non-iCloud Drive data storage. While it is possible it could be run
two-way research is needed on whether you’d have to download the old
file before you replace it.
- Timeouts may happen. iCloud Drive still gets stuck sometimes and there
will be non-zero time during downloads that don’t get stuck.
- There is no guarantee there is ever enough room on disk to hold a
specific file from iCloud Drive. This is likely resolved by MacOS
directly, however, this may take excessive time.
- If you thought running with checksums was slow before, strap in,
because those downloads are going to add up.
All that said, I think this would accomplish a high percentage of what
users are looking for.
Is this kind of a patch welcome? Is the plan sound? I haven't written C
since college (in the 90s!) and would be open to advice/mentoring.
Thank you.
regards,
bex
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
