[GitHub] [airflow] saguziel commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option

2020-02-25 Thread GitBox
saguziel commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option URL: https://github.com/apache/airflow/pull/7269#issuecomment-591116708 Celery does not support this type of thing, and it would not be portable across executors. Celery's consistency model is really simple, which

[GitHub] [airflow] saguziel commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option

2020-02-27 Thread GitBox
saguziel commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option URL: https://github.com/apache/airflow/pull/7269#issuecomment-592382747 DB load. One example is from like from 9% to 6% load.. but this is very dependent on the configs used. It would reduce writes by 30-40% on our

[GitHub] [airflow] saguziel commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option

2020-02-28 Thread GitBox
saguziel commented on issue #7269: [AIRFLOW-6651] Add Redis Heartbeat option URL: https://github.com/apache/airflow/pull/7269#issuecomment-592768775 BTW, this code is complex, but the complexity is stuck inside a single family of classes, which means it should not pollute the rest of the co