GitHub user raphaelauv added a comment to the discussion: Dupcating BigQuery job when using BigQueryInsertJobOperator with deferrable
yes you can contribute to the operator to make it check if there is already a job running or finish with exactly the same labels that are unique to every airflowtask ( kind of what is doing the KubernetesPodOperator ) to reduce the risk of running a duplicated query but in all case you should make your query idempotent GitHub link: https://github.com/apache/airflow/discussions/51073#discussioncomment-13281510 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
