kacpermuda commented on PR #35796:
URL: https://github.com/apache/airflow/pull/35796#issuecomment-1823184332

   Sure. The example DAG would be just simple execution of these operators, as 
their core function is send proper lineage events to the OpenLineage backend 
and not to perform any actions within a service (S3 in this case). Simpliest 
possible DAG would be:
   ```
   from airflow.models.dag import DAG
   
   from airflow.providers.amazon.aws.operators.s3 import (
       S3DeleteObjectsOperator,
       S3CreateObjectOperator,
       S3CopyObjectOperator,
   )
   from airflow.utils.dates import days_ago
   
   BUCKET = "testbucket"
   
   with DAG(
       dag_id="s3_test",
       schedule_interval=None,
       start_date=days_ago(2),
   ) as dag:
       create_file = S3CreateObjectOperator(
           task_id="create_file",
           dag=dag,
           s3_bucket=BUCKET,
           s3_key="a.csv",
           data="test1",
       )
       copy_file = S3CopyObjectOperator(
           task_id="copy_file",
           dag=dag,
           source_bucket_name=BUCKET,
           source_bucket_key="a.csv",
           dest_bucket_key=f"s3://{BUCKET}/c.csv",
       )
   
       delete_file = S3DeleteObjectsOperator(task_id="delete_file", dag=dag, 
bucket=BUCKET, keys=["a.csv", "b.csv"])
       create_file >> copy_file >> delete_file
   
   ```
   
   These methods that I implemented would not be explicitly called anywhere in 
`execute` or by the user. They are part of OpenLineage and Airflow integration 
(similar examples: #35778 or #35660 ) so they would be called just before 
`execute` (for `_on_start` method) and just after `execute` (for `_on_complete` 
method). If You have an OpenLineage backend configured, they would extract the 
lineage event that will be sent to the backend. 
[Here](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-53+OpenLineage+in+Airflow)
 is an AIP for OL integration.
   
   Please let me know if You need some additional information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to