[ https://issues.apache.org/jira/browse/AIRFLOW-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Imberman closed AIRFLOW-30. ---------------------------------- Resolution: Auto Closed > Make preoperators part of the same transaction as the actual operation > ---------------------------------------------------------------------- > > Key: AIRFLOW-30 > URL: https://issues.apache.org/jira/browse/AIRFLOW-30 > Project: Apache Airflow > Issue Type: Improvement > Components: operators > Reporter: Bence Nagy > Assignee: Kevin Mandich > Priority: Major > > All my use cases would work better if each operator would execute everything > in one transaction. Two examples: > - I want to {{GenericTransfer}} a set of rows from one DB to another, and I > have to create the table first in the destination DB. I feel like it'd be a > lot more clean if I didn't have empty tables lying around if the insertion > fails for some reason later on. > - I want to {{GenericTransfer}} all rows from an entire table periodically to > sync it from one DB to another. To do this correctly I want to clear the > destination table first to make sure I end up with no duplicate rows, so I'd > have a {{DELETE * FROM dst_table}} preoperator. If the insertions fail > afterwards, I'd end up with no data (it would be better in most cases to fall > back to the old data), and even if everything is working correctly, I'll have > an empty table while the insertions as still executing. > To fix this, the relevant {{DbApiHook}} methods could support a new kwarg to > set whether it should commit at the end. > Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005)