[ https://issues.apache.org/jira/browse/AIRFLOW-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894640#comment-16894640 ]
jack commented on AIRFLOW-3503: ------------------------------- [~Urbaszek] I think the difference is that you are deleing a file by specifying it directly while in his example he deletes a folder (note his function name and also the star in the end of the path. {code:java} gcs_export_uri = BQ_TABLE_NAME + '/' + EXEC_TIMESTAMP_PATH + '/*' {code} What happens to you if you put your file in a/file.txt and instead of specifying a/file.txt you specify a/* According to [~lot] report the first attempt needs to work (because file exist) and second should fail (because file does not exist) > GoogleCloudStorageHook delete return success when nothing was done > ------------------------------------------------------------------- > > Key: AIRFLOW-3503 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3503 > Project: Apache Airflow > Issue Type: Bug > Components: gcp > Affects Versions: 1.10.1 > Reporter: lot > Assignee: Yohei Onishi > Priority: Major > Labels: gcp, gcs, hooks > > I'm loading files to BigQuery from Storage using: > > {{gcs_export_uri = BQ_TABLE_NAME + '/' + EXEC_TIMESTAMP_PATH + '/*' > gcs_to_bigquery_op = GoogleCloudStorageToBigQueryOperator( dag=dag, > task_id='load_products_to_BigQuery', bucket=GCS_BUCKET_ID, > destination_project_dataset_table=table_name_template, > source_format='NEWLINE_DELIMITED_JSON', source_objects=[gcs_export_uri], > src_fmt_configs=\{'ignoreUnknownValues': True}, > create_disposition='CREATE_IF_NEEDED', write_disposition='WRITE_TRUNCATE', > skip_leading_rows = 1, google_cloud_storage_conn_id=CONNECTION_ID, > bigquery_conn_id=CONNECTION_ID)}} > > After that I want to delete the files so I do: > {{def delete_folder():}} > {{ """}} > {{ Delete files Google cloud storage}} > {{ """}} > {{ hook = GoogleCloudStorageHook(}} > {{ google_cloud_storage_conn_id=CONNECTION_ID)}} > {{ hook.delete(}} > {{ bucket=GCS_BUCKET_ID,}} > {{ object=gcs_export_uri)}} > > > {{This runs with PythonOperator.}} > {{The task marked as Success even though nothing was deleted.}} > {{Log:}} > [2018-12-12 11:31:29,247] \{base_task_runner.py:98} INFO - Subtask: > [2018-12-12 11:31:29,247] \{transport.py:151} INFO - Attempting refresh to > obtain initial access_token [2018-12-12 11:31:29,249] > \{base_task_runner.py:98} INFO - Subtask: [2018-12-12 11:31:29,249] > \{client.py:795} INFO - Refreshing access_token [2018-12-12 11:31:29,584] > \{base_task_runner.py:98} INFO - Subtask: [2018-12-12 11:31:29,583] > \{python_operator.py:90} INFO - Done. Returned value was: None > > > I expect the function to fail and return something like "file was not found" > if there is nothing to delete Or let the user decide with specific flag if he > wants the function to fail or success if files were not found. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)