YingboWang commented on issue #5499: [WIP][AIRFLOW-3964] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-538674356 @seelmann We are deploying the smart sensor inside our company right now. Testing and monitoring it for further refinement. I will update the code for this PR and AIP document with the overall structure and some statistics based on our testing for the past several weeks. @BasPH @feng-tao, Thanks for your feedbacks. It's my bad not update this PR on time. The smart sensor operator itself is only used to create a control dag for taking over all sensors in the airflow cluster when the cluster is enable to "use_smart_sensor" by configuration. The smart sensor task is running as a service keep refreshing all active sensor tasks from DB and poking them. Users doesn't need to change their original sensor tasks when writing DAGs. When use smart sensor, the sensor tasks state changing is different with regular sensor. The scheduler will persist sensor's poke context to DB and set the state to "Smart_pending" so that the task will not be queued and taking a separate process in any worker. Smart sensor task instance which is a long running process on the work will refresh DB to get all sensors that are in "Smart" state (smart_running or smart_pending) and poke them. If the poke return True it will set the corresponding task_instance state to "success" so that the scheduler will be able to kick off the downstream task_instance in the next parsing loop. There is only a cluster configuration change for using smart sensor. No need to write DAGs in a different way. I will update the PR and the description with latest change and also update the AIP to include all details for smart sensor design.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
