wjddn279 commented on issue #61453: URL: https://github.com/apache/airflow/issues/61453#issuecomment-3869484541
@pjavier29 Yes, in my opinion, considering the usage scenario, I don't think additional filtering is necessary (refer to PR description), but in-memory filtering would be nice. However, I have a question. What was the main cause of the performance degradation you observed? 1. Network IO from passing long string queries (the items going into the IN operation) 2. Query performance of the IN operation In case of 1, the same (or very similar) network IO would have occurred when calling https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2992. If 2 is the problem, doing in-memory filtering in Python would be slower than the database if anything, not faster, wouldn't it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
