Re: [I] ObjectStoragePath wrong template rendering [airflow]

[via GitHub]

astro-anand commented on issue #51909:
URL: https://github.com/apache/airflow/issues/51909#issuecomment-2996747054

   Hey @opeida and @gopidesupavan! The args to construct ObjectStoragePaths are 
not template fields. [Operators have template 
fields](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating),
 but ObjectStoragePaths are not operators. Rather, they're a utility for 
interacting with blob storage. You could update your code to something like.
   
   ```
   
   
   ...
       generate_uuid = PythonOperator(
           task_id="generate_uuid", python_callable=lambda: str(uuid4())
       )
   
       @task
       def get_stats_data(
           **context
       ) -> ObjectStoragePath | None:
           ...
           output_path = 
ObjectStoragePath(f"gs://{Variable.get('GCP_CONFIG')['GCS_BUCKET']}", 
conn_id="gcp-connection")
           output_path /= f"{context.get('ti').task_id}.csv"
           with output_path.open("w") as file:
               data.to_csv(file, index=False)
           return output_path
   ...
       stats_data = get_stats_data(output_path=base/generate_uuid.output)
   ```
   
   If you need to reuse the output path many times across multiple task, I'd 
suggest writing a function that returns the path to [avoid top level 
code](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code).
 Example usage below:
   
   ```
   ...
       generate_uuid = PythonOperator(
           task_id="generate_uuid", python_callable=lambda: str(uuid4())
       )
      
   def get_output_path():
          return 
ObjectStoragePath(f"gs://{Variable.get('GCP_CONFIG')['GCS_BUCKET']}", 
conn_id="gcp-connection")
   
       @task
       def get_stats_data(
           **context
       ) -> ObjectStoragePath | None:
           ...
           output_path = get_output_path()
           output_path /= f"{context.get('ti').task_id}.csv"
           with output_path.open("w") as file:
               data.to_csv(file, index=False)
           return output_path
   ...
       stats_data = get_stats_data(output_path=base/generate_uuid.output)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org