Flávio Clésio created AIRFLOW-2620:
--------------------------------------
Summary: New property called "RetainSameConnection"
Key: AIRFLOW-2620
URL: https://issues.apache.org/jira/browse/AIRFLOW-2620
Project: Apache Airflow
Issue Type: Wish
Components: configuration, operators, redshift
Affects Versions: Airflow 2.0
Reporter: Flávio Clésio
Some ETL stuff relies on the -high- reusability of the connections in the
middle of the data flow, and part of this is achieved using temporary tables.
I performed a test [1] in Airflow and the behaviour is the for each command
that we use the "PostgresOperator" hits the database again, and if the DAG had
a lot of connections it can be a nightmare.
As I said in SO, for several cases where you have cases of processing some
tables in stages the reuse a temporary table across multiple tasks can be very
useful and plus: can remove a lot of pressure in the RDBMS cause you won't need
to hit several times.
That's why a purpose a feature of the connection called "RetainSameConnection"
that will keep the same connection during the execution of the DAG.
Reference: [1] - [My question in Stack
Overflow|https://stackoverflow.com/questions/50858770/airflow-retain-the-same-database-connection/]
about it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
