1fanwang commented on PR #71250: URL: https://github.com/apache/airflow/pull/71250#issuecomment-5208527180
> Oh, fascinating > > Stefan, I am not familiar with TiDB. Doesn't have to be in response to this PR, but curious about your thoughts on why this for the Airflow meta database? Presumably for scaling, but more details would be very useful. Hey Vikram, I was actually just reading more about [[apache/airflow#46175](https://github.com/apache/airflow/issues/46175)](https://github.com/apache/airflow/issues/46175) and [https://github.com/apache/airflow/discussions/65453](https://github.com/apache/airflow/discussions/65453). To be forthcoming - I'm not proposing official support for Airflow on a TiDB backend, and this PR isn't meant to be a step toward asking for it, at least for now. Let me share 1. what I'm doing internally (Since some of this info is already public ([[1](https://www.youtube.com/watch?v=DzZ-I8WL2jM)](https://www.youtube.com/watch?v=DzZ-I8WL[2](https://www.youtube.com/watch?v=YFNS3PCvuQQ)jM), [2](https://www.youtube.com/watch?v=YFNS3PCvuQQ))) 2. and what I think is useful to the community today: For scale context, our largest single cluster is reaching 25k+ Dags and still growing. Most of the scaling problems we've hit have answers that stay close to upstream: add schedulers, tune the executor, etc.. The metadata DB is the one that doesn't. It's the piece you can't shard, since the scheduler critical section, TI state, XCom and event logs all land on one primary, and when that saturates the options are a bigger box (we've done that many times) or changes that drift away from OSS Airflow. I'd rather not drift. We've also been trying read/write splitting to take some pressure off it, since we already run read replicas. Either route reads explicitly in Airflow's own source, or put query routing rules in a [[ProxySQL](https://proxysql.com/)](https://proxysql.com/) layer so Airflow core can stay generic. No numbers to share yet, but the challenge is clear - it only moves read load, and the scheduler's hot path is writes. The queries that actually hurt are the ones you can't route away. That's what makes distributing writes interesting, and TiDB uses the MySQL wire protocol. I haven't benchmarked it though, so "scales writes" is a motivation and not a result I can show at this point. So far this is just a small local cluster with dev Airflow pointed at it, checking the SQL queries (the ones scheduler depends on). Most of it holds up: pessimistic FOR UPDATE blocks, NOWAIT errors, GET_LOCK is exclusive, READ COMMITTED behaves, savepoints roll back, FK cascades are enforced. The 3 prs I opened addresses some of the minor issues I found during testing. For this PR tho, my original thought is it isn't really a TiDB-specific problem. Any server that accepts SKIP LOCKED and quietly drops it hands two schedulers the same rows, and Airflow never finds out (it should fail loudly instead). `with_row_locks()` already degrades gracefully when a MySQL-family engine can't lock at all. I do plan on trying this on our internal Airflow and TiDB clusters. Happy to share what we find running Airflow on TiDB at scale if folks are interested. The three that came out of the exercise, for reference: - https://github.com/apache/airflow/pull/71249 - https://github.com/apache/airflow/pull/71250 - https://github.com/apache/airflow/pull/71251 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
