On Apr 14 2021, Grunthos <philip.john.war...@gmail.com> wrote: > On Wednesday, April 14, 2021 at 5:36:30 PM UTC+10 niko...@rath.org wrote: > >> >> Yes, all of these would be possible and probably be faster. I think >> option (2) would me the best one. >> >> Pull requests are welcome :-). >> >> > I had a funny feeling that might be the answer...and in terms of utility > and design, ISTM that " add a special s3ql command to do a 'tree copy' -- > it would know exactly which blocks it needed and download them en-masse > while restoring files (and would need a lot of cache, possibly even a > temporary cache drive)" is a good plan. > > I am not at all sure I am up for the (probable) deep-dive required, but if > I were to look at this could you give some suggested starting points? My > very naieve approach (not knowing the internals at all) would be to build a > list of all required blocks, do some kind of topo sort, then start multiple > download threads. As each block was downloaded, determine if a new file can > be copied yet, and if so, copy it, then release and blocks that are no > longer needed. > > ...like I said, naieve, and hightly dependant on internals...and maybe > should use some kind of private mount to avoid horror.
I think there's a simpler solution. 1. Add a new special xattr to trigger the functionality (look at s3qlcp.py and copy_tree() in fs.py) 2. Have fs.py write directly to the destination directory (which should be outside the S3QL mountpoint) 3. Start a number of async workers (no need for threads) that, in a loop, download blocks and write them to a given offset in a given fh. 4. Have the main thread recursively traverse the source and issue "copy" requests to the workers (through a queue) 5. Wait for all workers to finish. 6. Profit. I wouldn't even bother putting blocks in the cache - just download and write to the destination on the fly. It may be worth checking if a block is *already* in the cache and, if so, skip download though. With this implementation, blocks referenced by multiple files will be downloaded multiple times. I think this can be improved upon once the minimum functionality is working. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/874kg9l06t.fsf%40vostro.rath.org.