[GitHub] [airflow] ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#issuecomment-589645657 This PR may not be needed so much anymore once PRs such as #7477 land (this would still make _a bit_ of difference, but possibly not as much anymore.) I will re-investigate this in a week or two. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#issuecomment-575220751 Updated numbers from master branch. I'm not sure these will equate to _such_ a speed up in terms of actual task execution. Overall numbers are lower as I have a faster machine to run tests on now :) Concurrent DagRuns | Tasks | Before | After | Speedup -- | -- | -- | -- | -- 100 | 12 | 3.9545s (±0.140s) | 1.7174s (±0.045s) | x2.3 100 | 40 | 12.1224s (±0.357s) |9.0649s (±0.603s) | x1.3 1000 | 12 | 17.7790s (±0.240s) | 8.3937s (±0.516s) | x1.5 1000 | 40 | 67.1121s (±0.799s) | 44.6126s (±0.518s) | x1.5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#issuecomment-574885270 Tests aren't quite passing -- one failure due to a non-fixed ordering of return values. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#issuecomment-574766407 Tests should be passing now, and we've been running a back-ported version of this for a few weeks and it makes a difference for us. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#issuecomment-573258461 Hi @shlomiken Sorry, my inbox is a disaster area right now :) I've fixed that issue, and a few more. Something is still up with the tests here on master that I need to dig in to, but the backported version of this change to 1.10.5 should be ready tomorrow/Monday: https://travis-ci.com/astronomer/airflow/builds/14396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
