vincbeck commented on PR #73015: URL: https://github.com/apache/airflow/pull/73015#issuecomment-5819866637
> I really appreciate this PR in theory and love the speed increase the data shows. But I've been reading it and thinking that it might be a little sharp in ways the users don't expect. Specifically the fallback guarantee. It only holds if the prefix files and the tar are kept in sync, and this really seems to me like one of the pieces that will absolutely get out of sync without extreme diligence from the user. I know it's documented, but just seems like it might cause more harm and confusion. > > What about optimizing with the use of client.get_object() to avoid the two request per file and then also using some parallelism to download the objects concurrently to reduce it down further. I think that would get almost all of the same speed gains, but without the sharp edges of the duality this PR has. Either that, or have no fallback to the prefix if you use the archive, essentially make them mutually exclusive so that keeping them in sync isn't critical. > > CC @ramitkataria @vincbeck thoughts? I agree, I would actually do both. Optimize the current mechanism and add the archive option without fallback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
