Re: [PR] Add OpenLineage support to S3Operators - Copy, Delete and Create Object [airflow]

2023-11-22 Thread via GitHub
o-nikolas merged PR #35796: URL: https://github.com/apache/airflow/pull/35796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

Re: [PR] Add OpenLineage support to S3Operators - Copy, Delete and Create Object [airflow]

2023-11-22 Thread via GitHub
vincbeck commented on PR #35796: URL: https://github.com/apache/airflow/pull/35796#issuecomment-1823275031 Got it! Thanks for the details :) That makes more sense now. Code looks good to me -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Add OpenLineage support to S3Operators - Copy, Delete and Create Object [airflow]

2023-11-22 Thread via GitHub
kacpermuda commented on PR #35796: URL: https://github.com/apache/airflow/pull/35796#issuecomment-1823184332 Sure. The example DAG would be just simple execution of these operators, as their core function is send proper lineage events to the OpenLineage backend and not to perform any action

Re: [PR] Add OpenLineage support to S3Operators - Copy, Delete and Create Object [airflow]

2023-11-22 Thread via GitHub
vincbeck commented on PR #35796: URL: https://github.com/apache/airflow/pull/35796#issuecomment-1823071373 Would you have an example DAG which uses these new functions? Since the code you introduce is never executed in the `execute` method, I am wondering how that would look like in a DAG

[PR] Add OpenLineage support to S3Operators - Copy, Delete and Create Object [airflow]

2023-11-22 Thread via GitHub
kacpermuda opened a new pull request, #35796: URL: https://github.com/apache/airflow/pull/35796 Adding OpenLineage support for the following operators: - S3CopyObjectOperator - S3CreateObjectOperator - S3DeleteObjectsOperator --- **^ Add meaningful d