hussein-awala opened a new pull request, #36504:
URL: https://github.com/apache/airflow/pull/36504

   Airflow Metadata is the brain of Airflow; it is the central component used 
to store the state of all operations and enable communication between other 
components.
   
   A lot of the execution time of Airflow components is wasted waiting for some 
input of this Metadata, which we can avoid if we query the database 
asynchronously. We can perform other operations during the waiting time or send 
more requests to Metadata.
   
   SQLAlchemy supports since 1.4 [Asynchronous I/O 
(asyncio)](https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html) that 
we can use to improve all our components:
   - improving scheduler logic
   - improving the async execution in the executors
   - making the Triggerer fully asynchronous
   - loading variables/connections asynchronously in the triggers and the async 
hooks.
   - and, of course, improving the performance of our API and webserver, which 
will serve much more users with lower resource usage.
   
   To test and validate the efficiency of async sessions before merging 
complicated methods from sync to async, I implemented the utils to create the 
async sessions and some async methods to load the variables and the connections 
from the database, and I tried  one of the newly implemented methods with the 
triggerer:
   
   test_trigger.py:
   ```python
   from typing import Any
   
   from airflow.triggers.base import BaseTrigger
   
   
   class TestTrigger(BaseTrigger):
   
       def serialize(self) -> tuple[str, dict[str, Any]]:
           return "airflow.test_trigger.TestTrigger", {}
   
       async def run(self):
           import asyncio
   
           from airflow.models.connection import Connection
   
           for _ in range(1000):
               conn = await 
Connection.async_get_connection_from_secrets("airflow_db")
               self.log.info(conn.login)
               await asyncio.sleep(0.1)
           yield {}
   ```
   dag.py:
   ```python
   from datetime import datetime
   from airflow.models.dag import DAG
   from airflow.models.operator import BaseOperator
   from airflow.test_trigger import TestTrigger
   
   
   class TestOperator(BaseOperator):
       def __init__(self, param: int, **kwargs):
           super().__init__(**kwargs)
           self.param = param
   
       def execute(self, context):
           self.defer(
               trigger=TestTrigger(),
               method_name="execute_complete",
           )
   
       def execute_complete(self):
           self.log.info("Done!")
   
   
   with DAG(
       dag_id="async_db",
       schedule_interval=None,
       start_date=datetime(2023, 12, 31),
       catchup=False,
   ) as dag:
       TestOperator.partial(task_id="test_task").expand(param=list(range(1000)))
   ```
   I triggered a manual Dag Run, which created 1000 mapped tasks and deferred 
them.
   
   I had the expected result in the log (login root) without any blocking in 
the event loop of the Triggerer, with reasonable resource consumption.
   
   Then I tested by replacing the trigger run method with:
   ```python
       async def run(self):
           import asyncio
   
           from airflow.models.connection import Connection
   
           for _ in range(1000):
               conn = Connection.get_connection_from_secrets("airflow_db")
               self.log.info(conn.login)
               await asyncio.sleep(0.1)
           yield {}
   ```
   I got the result in the task log, but the Triggerer log was full of:
   ```
   [2023-12-31T01:42:36.934+0000] {triggerer_job_runner.py:598} INFO - trigger 
async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/235/1 (ID 471) 
starting
   [2023-12-31T01:42:37.091+0000] {triggerer_job_runner.py:598} INFO - trigger 
async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/236/1 (ID 472) 
starting
   [2023-12-31T01:42:37.159+0000] {triggerer_job_runner.py:576} INFO - 
Triggerer's async thread was blocked for 0.57 seconds, likely by a 
badly-written trigger. Set PYTHONASYNCIODEBUG=1 to get more information on 
overrunning coroutines.
   [2023-12-31T01:42:37.369+0000] {triggerer_job_runner.py:598} INFO - trigger 
async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/237/1 (ID 473) 
starting
   [2023-12-31T01:42:37.549+0000] {triggerer_job_runner.py:598} INFO - trigger 
async_db/manual__2023-12-31T01:42:07.645143+00:00/test_task/239/1 (ID 474) 
starting
   [2023-12-31T01:42:37.772+0000] {triggerer_job_runner.py:576} INFO - 
Triggerer's async thread was blocked for 0.61 seconds, likely by a 
badly-written trigger. Set PYTHONASYNCIODEBUG=1 to get more information on 
overrunning coroutines.
   ```
   and the Triggerer used almost all the resources of my computer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to