henry3260 commented on PR #74119: URL: https://github.com/apache/airflow/pull/74119#issuecomment-5969997938
> My one design decision concern before addressing the claude review: > > Would it be better to validate the graph at once "after the Dag be registered" so that we don't need to traverse whole graph again whenever a new edge be added? > > Just like the TS side that we will only validate the cycle at the "finalize stage". I think we should do the same "last mile validation right before the serve" instead of having iterative validation (since we will compile the SDK into artifact anyway) Agreed, done in the latest push. The cycle check moved into markRegistered, next to the existing Then check, so it runs once over the whole graph instead of once per edge: O(V+E) rather than O(E*(V+E)). I rebuilt your reverse-declared chain locally, 2,000 tasks in 9ms and 8,000 in 24ms end to end (building the Dag plus the Register walk), against the 0.19s/2.1s you measured. Different machine, but it is linear now. This also removes the need for the rollback in your other comment: recording an edge no longer panics, so a rejected fan-out leaves the Dag exactly as it was, and the caveat in the Before doc is gone with it. A task ordered against itself is still rejected where it is declared, since that needs no walk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
