I think yes - as long as we avoid matrix testing and a complicated release schedule, it's good.
J On Wed, May 13, 2026 at 9:04 PM Tzu-ping Chung <[email protected]> wrote: > OK, I think at this point we are both pretty determined on certain things > on this. Let’s see if there’s a compromise. > > The things I absolutely want are > > 1. The coordinator to be defined under airflow.sdk.coordinators and a > public interface. I don’t want the possibility of needing to change package > name in the future. > 2. The configuration to use the package path as a public interface. Not > mentioning the coordinator identifier would require each future coordinator > having its own configuration, or needing to change the configuration > format. Both require much deprecation work. > > I can stand the coordinator can be released as a part of Task SDK > initially (under the aforementioned package name). Even if it happens > earlier, user education shouldn’t be TOO bad…? As long as the import path > stays the same, this is a relatively simple fix in most deployments. The > worst case is we only split in Airflow 4. > > Would this be acceptable from your perspective Jarek? > > TP > > > On 14 May 2026, at 02:21, Jarek Potiuk <[email protected]> wrote: > > Well. in this case you have just one class. one coordinator and even fo > other (non-java) coordiatonr the "classpath" is wrong thing to say so. You > could achieve the same by not stating the classpath - but simply stating > which java interpreters to use. > > [sdk] > >> jdk_bridge = { > >> "jdk-11": { > >> "kwargs": {"java_executable": > "/usr/lib/jvm/java-11-openjdk/bin/java", "jars_root": ["/files/old/lib"]} > >> }, > >> "jdk-17": { > >> "kwargs": {"java_executable": > "/usr/lib/jvm/java-17-openjdk/bin/java", "jars_root": ["/files/new/lib"], > "jvm_args": ["-Xmx1024m"]} > >> } > >> } > > > I really don’t understand the desire to have the Java coordinator inside > the Task SDK distribution in the first go. The coordinator class must be > public in the worker (at least the import path), and putting it in the SDK > does not provide any more freedom to change it faster. It’s the contrary > because Task SDK releases require significantly more testing since the > distribution contains many things, while providers (in a similar position > to Airflow Core as coordinators to Task SDK) are released more frequently, > and can have major version bumps on their own if needed. > > If we agree that you want to release bugfixes for the task SDK > independently and faster, then yes, separate distribution might be a good > reason. But you need to solve the SDK's version coupling issue to make it > happen. > > his introduces operational complexities - depending on what kind of > version coupling you choose between SDK and coordinators. > > > What is the versioning and compatibility scheme you see? That will > significantly impact testing complexity and the release schedule - because > we will have to maintain a parallel release "train" for the "coordinator". > For example when a new SDK coordinator is released, it must work with > existing SDKs—imagine we have SDK 1.2.*, 1.3.*, 1.4.*, 1.5.*. Will the new > version of task-sdk be compatible? Should we add back-compat tests for all > those versions?) . And I am not even talking about intentionally breaking > the APIs, but unintentional bugs. Also if someone uses the new version of > the "SDK" but doesn't update the old version of the "Java Coordinator." > Will that continue to work? How do we ensure that? Are we going to test all > SDK versions with all "coordinator" versions? > > This is the operational complexity I am talking about. We already have > this for providers and it only works because we intentionally limited > back-compatibility and we run all those tests for older airflow versions > and we have "year" stable and proven BaseHook and BaseOperator API that has > not changed for years after it stabilized. > And we could limit that operational complexity - for example by coupling > minor versions. For example we say SDK 1.2.* Only works with coordinator > 1.2. *, SDK 1.3.* Only works with coordinator 1.3.* Assuming only bugfixes > are done in each. That also means that coordinator changes from main will > need to be cherry picked to v3_N_test and the faster releases of > coordinators will have to be released from v3_N_stable branch - that will > limit back-compat tests, but increases the development complexity - because > you will have to cherry-pick changes and have - potentially - independent > releases of coordinator 1_N from that branch where it will be tested with > Airflow 3.N and SDK 3.N. > > So we have those trade-offs: > > 1) Strict coupling (pinning) SDK version == Coordinator version: Slower > bug fix cycle, but no back-compat testing needed > 2) Coupling SDK MAJOR.MINOR = Coordinator MAJOR.MINOR => Faster bugfix > cycles, increased development/release complexity, leading to cherry-picking > to the v3_branch and separate releases for the coordinator from that > branch, back-compatibility testing is limited to that v3_N_test branch > 3) Free-fall: Any SDK works with Any Coordinator => faster bugfix cycles, > simpler releases and development (releases done from main) -> hugely > complex matrix of compatibility tests that might slow down testing even more > > There is also a fourth option: what we do for providers which is "limited > free form." We deliberately limit "min_version" in providers and bump it > regularly to reduce our compatibility matrix size. > > Those are basically the three choices we have. I personally think option 1 > is best at this stage. We release the task-sdk with Airflow every month. > When needed, and if we find a critical bug we can do an ad-hoc release. > > Which one would you prefer - and do you also want to commit to maintaining > the associated development/testing complexity if it is not 1) ? > > J. > > > > > > > On Wed, May 13, 2026 at 7:07 PM Tzu-ping Chung via dev < > [email protected]> wrote: > >> You can do the same if it’s in the task sdk, but >> >> 1. You need to use the same import path, but then you need to separately >> teach users to install a new package before moving it out. Not a very good >> user experience. >> >> 2. Or you use a different import path. You need to keep the old path >> working in the distribution for a long time *and* have users change their >> configs to fix the deprecation warning (and eventual breaking change). >> Unnecessary mental gymnastics on both sides. >> >> I really don’t understand the desire to have the Java coordinator inside >> the Task SDK distribution in the first go. The coordinator class must be >> public in the worker (at least the import path), and putting it in the SDK >> does not provide any more freedom to change it faster. It’s entirely the >> contrary since Task SDK releases require a lot more testing since the >> distribution contains many things, while providers (in a similar position >> to Airflow Core as coordinators to Task SDK) are released more frequently, >> and can have major version bumps on their own if needed. >> >> >> > On 14 May 2026, at 00:54, Jarek Potiuk <[email protected]> wrote: >> > >> > > You can create multiple instances of the same coordinator class. Pass >> appropriate arguments to suite your need. This is in the AIP. >> > >> > Yes. And you cand do exactly the same 1-1 if it's part of package and >> embedded in "airflow-sdk" distribution? Or am I wrong? Why do you think it >> would not be possible if it's part of task_sdk? >> > >> > On Wed, May 13, 2026 at 6:43 PM Tzu-ping Chung via dev < >> [email protected] <mailto:[email protected]>> wrote: >> >> You can create multiple instances of the same coordinator class. Pass >> appropriate arguments to suite your need. This is in the AIP. >> >> >> >> [sdk] >> >> coordinators = { >> >> "jdk-11": { >> >> "classpath": "airflow.sdk.coordinators.java.JavaCoordinator", >> >> "kwargs": {"java_executable": >> "/usr/lib/jvm/java-11-openjdk/bin/java", "jars_root": ["/files/old/lib"]} >> >> }, >> >> "jdk-17": { >> >> "classpath": "airflow.sdk.coordinators.java.JavaCoordinator", >> >> "kwargs": {"java_executable": >> "/usr/lib/jvm/java-17-openjdk/bin/java", "jars_root": ["/files/new/lib"], >> "jvm_args": ["-Xmx1024m"]} >> >> } >> >> } >> >> >> >> The problem is, classpath points to a class, so whatever this string >> is needs to be kept compatible in future releases. >> >> >> >> >> >> >> >> > On 13 May 2026, at 23:59, Jarek Potiuk <[email protected] <mailto: >> [email protected]>> wrote: >> >> > >> >> > > How do you make it changeable any time? User needs to be able to >> specify what coordinator to use in the config, and you can’t break that >> later. >> >> > We have only *jdk* coordinator now . So I will revert the question. >> How are you going to configure two "jdk" `coordinators` when you have >> separate distributions running "java"? Are you planning to install two >> "coordinator-jdk" packages? This isn't possible in Python unless you build >> almost the same package with jdk-11, jdk-19 built in? >> >> > My understanding is that you will have configuration options to >> choose between "jdk-11" and "jdk-19." This "jdk" package of yours will >> simply have a list of "jdks" linking to the Java interpreters. >> >> > >> >> > So, it doesn't matter if it's a single "coordinator-jdk" package or >> everything in "airflow.sdk._coordinator."jdk" package or >> "airflow.sdk._bridge.jdk" package in the task-sdk. Regardless, you cannot >> install two "jdk" packages, whether they are separate distributions or if >> the package is in "task-sdk" and you have to configure which of the "jdk" >> bridges you want to use. >> >> > Yes. Sometime later, when we also have Go/TypeScript or other >> languages, we might decide to centralize some APIs, create a "true" >> coordinator package and separate distributions. And splitting into >> different packages will be absolutely no problem then. Nothing will stop us >> from doing it. >> >> > >> >> > It will save a lot of time for all the "distribution" issue - >> including releases, packagig, CI and everything connected - and it does not >> absolutely block us from further split later. >> >> > >> >> > J. >> >> > >> >> > >> >> > >> >> > On Wed, May 13, 2026 at 3:47 PM Tzu-ping Chung via dev < >> [email protected] <mailto:[email protected]> <mailto: >> [email protected] <mailto:[email protected]>>> wrote: >> >> >> >> >> >> >> >> >> > On 13 May 2026, at 20:42, Jarek Potiuk <[email protected] <mailto: >> [email protected]> <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> >> >> > >> >> >> > Not really. I proposed an internal package that can be changed >> **any time**. Users aren't supposed to use those items. We can clearly mark >> them with "_" and also describe them thoroughly in the public API >> documentation. And no. Initilaly providers were **not** in arflow at all - >> you started from step 2. Step 1 is that they were added at some point in >> time long before my time. Hooks and operators as "API" were creaed quite >> early in the concept of Airflow - and the first implementations were added >> then. Then, after common patterns emerged, those hooks and operators were >> grouped into providers (they were not initially) and only moved out after >> quite some time. As I see it - you even admit yourself that things will >> look differently for different languages, and maybe even we will not need >> bridges for some of them at all. So why should we introduce new concept if >> we know currrently that it applies only to "JDK"? I fail to see why we >> should proceed if we already know the patterns are unlikely to be reusable >> in their current form. >> >> >> >> >> >> How do you make it changeable any time? User needs to be able to >> specify what coordinator to use in the config, and you can’t break that >> later. >> >> >> >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> To unsubscribe, e-mail: [email protected] <mailto: >> [email protected]> <mailto: >> [email protected] <mailto: >> [email protected]>> >> >> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> >> >> > > >
