SuperBo opened a new issue, #37007:
URL: https://github.com/apache/airflow/issues/37007
### Apache Airflow version
2.8.1
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When using DatabricksSQLOperator with csv as output format, Operation failed.
Log
```
[2024-01-25, 08:48:30 UTC] {client.py:258} INFO - Closing session
01eebb5e-8163-1e9c-a85e-6d23051ebfea
[2024-01-25, 08:48:30 UTC] {taskinstance.py:2698} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py",
line 433, in _execute_task
result = execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/common/sql/operators/sql.py",
line 296, in execute
return self._process_output([output], hook.descriptions)[-1]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/operators/databricks_sql.py",
line 149, in _process_output
writer.writerow(row.asDict())
^^^^^^^^^^
AttributeError: 'Row' object has no attribute 'asDict'
[2024-01-25, 08:48:30 UTC] {taskinstance.py:1138} INFO - Marking task as
FAILED. dag_id=example_dag, task_id=select_account_info,
execution_date=20240125T072737, start_date=20240125T084825,
end_date=20240125T084830
[2024-01-25, 08:48:30 UTC] {standard_task_runner.py:107} ERROR - Failed to
execute job 61 for task select_account_info ('Row' object has no attribute
'asDict'; 391)
[2024-01-25, 08:48:30 UTC] {local_task_job_runner.py:234} INFO - Task exited
with return code 1
[2024-01-25, 08:48:30 UTC] {taskinstance.py:3280} INFO - 0 downstream tasks
scheduled from follow-on schedule check
```
### What you think should happen instead?
It shouldn't raise error.
### How to reproduce
Create following operator
```python
select_account_info = DatabricksSqlOperator(
task_id="select_account_info",
sql="SELECT a, b, c FROM tabl",
output_format="csv",
output_path="/var/tmp/output.csv",
)
```
### Operating System
MacOS
### Versions of Apache Airflow Providers
apache-airflow-providers-databricks==6.0.0
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]