jason810496 commented on issue #69288: URL: https://github.com/apache/airflow/issues/69288#issuecomment-5065728146
Hi @guan404ming, @shivaam, I think we can start discussing the user interface for the native TypeScript Dag definition here, if you'd like to join. Otherwise, I'll start the design on my side. (No need to start implementing at this point, just a discussion about what exactly the interface will look like.) The high-level idea of how the native Dag works for each Lang-SDK is: - We first define a DSL for the Dag and Task (we should be able to set, for example,` schedule="@daily"` at the Dag level, `retries=3` at the Task level, etc.). - Then the Lang-SDK runtime should be able to return an `airflow-core/src/airflow/serialization/schema.json`-compatible Dag JSON back to the coordinator via the same comm protocol. Some points we need to be aware of: - From a language perspective, whether the syntax itself can feasibly return a serialized Dag JSON. - Whether the syntax is "TaskFlow"-like enough. By TaskFlow, I mean users can define dependencies through "chaining," without needing something like a "taskId as a string/array" reference on the other task. - How arguments get resolved in the TS-SDK runtime across tasks in the same language, and for the same task across different languages. For reference, here are the Go-SDK counterparts: - https://github.com/apache/airflow/pull/70158 -- Go-SDK native Dag syntax (no need to dive into the implementation; the syntax in the PR description is enough). - https://github.com/apache/airflow/pull/69757 -- the foundation for resolving arguments across native Dags and across different languages: we serialize the argument type annotation as a JSON schema and store it in the serialized Dag. Either way, let's focus only on the user interface design for now. Please let me know WDYT, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
