hankehly opened a new pull request, #30981:
URL: https://github.com/apache/airflow/pull/30981

   Related: #22303
   
   This PR proposes a change to the 
[`PubSubHook.publish`](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/hooks/pubsub/index.html#airflow.providers.google.cloud.hooks.pubsub.PubSubHook.publish)
 method. I updated the method signature to include support for string-type 
message data. Previously, the operator only allowed byte-type message data, 
which made it complicated for users to use templated message data. For example, 
the following would not work.
   
   ```py
   with DAG(...):
       @task
       def create_messages():
           return json.dumps([{"data": "hello world"}])
   
       publish_messages = PubSubPublishMessageOperator(
           task_id="publish_messages",
           project="test",
           topic="test",
           messages="{{ ti.xcom_pull('create_messages') | to_bytes }}"
       )
   
       create_messages() >> publish_messages
   ```
   
   Since bytes-type data is not JSON serializable, the only way to make the 
above example work is to set 
[`render_template_as_native_obj=True`](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#rendering-fields-as-native-python-objects)
 alongside a custom jinja2 filter to convert the xcom value to bytes.
   
   ```py
   def to_bytes(value):
       return value.encode("utf-8")
   
   with DAG(..., render_template_as_native_obj=True, 
user_defined_filters={"to_bytes": to_bytes}):
       @task
       def create_messages():
           return json.dumps([{"data": "hello world"}])
   
       publish_messages = PubSubPublishMessageOperator(
           task_id="publish_messages",
           project="test",
           topic="test",
           messages="{{ ti.xcom_pull('create_messages') | to_bytes }}"
       )
   
       create_messages() >> publish_messages
   ```
   
   With this change, users can now easily use templated message data in string 
format, making the operator more versatile and user-friendly. The changes are 
backward compatible, so existing operators can continue to use bytes-type 
message data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to