SuperBo opened a new issue, #37007:
URL: https://github.com/apache/airflow/issues/37007

   ### Apache Airflow version
   
   2.8.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   When using DatabricksSQLOperator with csv as output format, Operation failed.
   
   Log
   ```
   [2024-01-25, 08:48:30 UTC] {client.py:258} INFO - Closing session 
01eebb5e-8163-1e9c-a85e-6d23051ebfea
   [2024-01-25, 08:48:30 UTC] {taskinstance.py:2698} ERROR - Task failed with 
exception
   Traceback (most recent call last):
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py",
 line 433, in _execute_task
       result = execute_callable(context=context, **execute_callable_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/common/sql/operators/sql.py",
 line 296, in execute
       return self._process_output([output], hook.descriptions)[-1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File 
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/databricks/operators/databricks_sql.py",
 line 149, in _process_output
       writer.writerow(row.asDict())
                       ^^^^^^^^^^
   AttributeError: 'Row' object has no attribute 'asDict'
   [2024-01-25, 08:48:30 UTC] {taskinstance.py:1138} INFO - Marking task as 
FAILED. dag_id=example_dag, task_id=select_account_info, 
execution_date=20240125T072737, start_date=20240125T084825, 
end_date=20240125T084830
   [2024-01-25, 08:48:30 UTC] {standard_task_runner.py:107} ERROR - Failed to 
execute job 61 for task select_account_info ('Row' object has no attribute 
'asDict'; 391)
   [2024-01-25, 08:48:30 UTC] {local_task_job_runner.py:234} INFO - Task exited 
with return code 1
   [2024-01-25, 08:48:30 UTC] {taskinstance.py:3280} INFO - 0 downstream tasks 
scheduled from follow-on schedule check
   
   ```
   
   ### What you think should happen instead?
   
   It shouldn't raise error.
   
   ### How to reproduce
   
   Create following operator
   
   ```python
   select_account_info = DatabricksSqlOperator(
       task_id="select_account_info",
       sql="SELECT a, b, c FROM tabl",
       output_format="csv",
       output_path="/var/tmp/output.csv",
   )
   ```
   
   ### Operating System
   
   MacOS
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-databricks==6.0.0
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to