[PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-30 Thread via GitHub
hussein-awala opened a new pull request, #36504: URL: https://github.com/apache/airflow/pull/36504 Airflow Metadata is the brain of Airflow; it is the central component used to store the state of all operations and enable communication between other components. A lot of the execution

Re: [PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-30 Thread via GitHub
hussein-awala commented on PR #36504: URL: https://github.com/apache/airflow/pull/36504#issuecomment-1872653755 For the drivers: - SQLite: - https://github.com/omnilib/aiosqlite - PostgreSQL: - https://magicstack.github.io/asyncpg/current/ - https://www.psycopg.org/docs/a

Re: [PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-31 Thread via GitHub
potiuk commented on PR #36504: URL: https://github.com/apache/airflow/pull/36504#issuecomment-1872889631 I expect a long time to go/test all cases . But I love it :). But yeah, if we limit it for Connections and Variables for now, that would be cool. I think one general comment here i

Re: [PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-31 Thread via GitHub
potiuk commented on PR #36504: URL: https://github.com/apache/airflow/pull/36504#issuecomment-1872896761 Also - I tihnk there is a completely separate question on what we do with "secrets" . I guess this sync/async configuration should also apply to secrets and we should extend our Secrets

Re: [PR] [POC][WIP] Async SQLAlchemy sessions in Airflow [airflow]

2023-12-31 Thread via GitHub
hussein-awala commented on PR #36504: URL: https://github.com/apache/airflow/pull/36504#issuecomment-1872955940 > I expect a long time to go/test all cases . But I love it :). > > But yeah, if we limit it for Connections and Variables for now, that would be cool. I think one general c