MannXo commented on issue #70953: URL: https://github.com/apache/airflow/issues/70953#issuecomment-5163396231
Tested my three changes in this RC round. All three behave as intended. Setup: `apache-airflow==3.3.0` in a clean venv on Python 3.12, one provider RC installed per check, real backends in Docker (MongoDB 7, MinIO for S3, Weaviate 1.28.4). Each case is a real DAG run via `airflow dags test`, with `render_template_as_native_obj=True` and the templated field pulled from an upstream XCom, since all three fixes are about *when* a template field is inspected. For each one I also ran the same DAG against the previous release to confirm the RC actually changes the behaviour. --- **`apache-airflow-providers-amazon==9.34.0rc1`, [#70330](https://github.com/apache/airflow/pull/70330) (MongoToS3Operator)** `mongo_query` rendering to a list now takes the aggregate path: ``` CASE 1: mongo_query renders to a list -> aggregate pipeline s3://rc-bucket/pipeline.json -> {"_id": "red", "total": 7} CASE 2: mongo_query renders to a dict -> find s3://rc-bucket/find.json -> {"_id": 1, ...} | {"_id": 2, ...} ``` The `$group` result can only come from `aggregate`, so that pins the path. On 9.33.0 the same DAG fails, because `is_pipeline` was decided in `__init__` from the un-rendered string and the rendered list went to `find`: ``` TypeError: filter must be an instance of dict, bson.son.SON, or any other type that inherits from collections.Mapping ``` **`apache-airflow-providers-cohere==1.6.7rc1`, [#70325](https://github.com/apache/airflow/pull/70325) (CohereEmbeddingOperator)** I have no Cohere API key, so the connection's `base_url` points at a local stub of the `/v2/embed` endpoint. The operator and the Cohere SDK are real, only the remote is not, which is enough to see the exact payload that goes out: ``` CASE 1: input_text renders to a single string -> POST /v2/embed texts=["a single sentence"] CASE 2: input_text renders to a list of strings -> POST /v2/embed texts=["first sentence", "second sentence"] ``` On 1.6.6 case 2 sends a nested list, since `__init__` wrapped the un-rendered `"{{ ... }}"` string and templating then filled the list element in: ``` texts=[["first sentence", "second sentence"]] ``` **`apache-airflow-providers-weaviate==3.4.1rc1`, [#70326](https://github.com/apache/airflow/pull/70326) (WeaviateIngestOperator)** ``` CASE 1: input_data templated from XCom, renders to a list of objects objects in weaviate: ['alpha', 'beta'] CASE 2: the same field renders to None TypeError: input_data is required (task failed, nothing ingested) ``` On 3.4.0 case 2 gets `TypeError: 'NoneType' object is not iterable` from inside `WeaviateHook.batch_data` instead. Same failure, clearer message, and constructing the operator with `input_data=None` no longer raises at DAG parse time. --- No other issues seen with these three distributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
