shahar1 commented on issue #11323: URL: https://github.com/apache/airflow/issues/11323#issuecomment-3858820955
> There are cases where extreme misuse isn't defended against, and defensive logic and exception branches can make the code harder to maintain. Still, users who are already using the operator in a "misuse" way (e.g. bucket-to-bucket transfer with many files) could suddenly run into the XCom limit once we start returning destination URIs. In my view, if those users are to be considered, something like a parameter named `return_destination_uris` with a default of `False` could be one option. It would keep the default safer (no large return value) while still allowing callers who need the list to opt in. Personally, I don't think that we should be overly gentle with extreme misuse. XComs are inherently a metadata mechanism, and if the list of files is overly large - the users should adopt the appropriate patterns (e.g., using a custom XCom backend or migrating to Google-cloud based transfer operation). As I mentioned, the option to avoid pushing to xcoms already exists, and we're already do our best to implement it in a slow backward-comaptibile way. We shouldn’t design defaults around anti-patterns that go against Airflow best practices. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
