jason810496 opened a new issue, #66451:
URL: https://github.com/apache/airflow/issues/66451
## Why
Question from the community:
- How can I use different JDK version?
- How can I use different JVM arguments?
We hardcoded the subprocess cmd as the following, so user have to subclass
another Coordinator to override the Java config to solve the above questions.
```python
return [
"java",
"-classpath",
classpath,
BundleScanner.resolve_jar(jar_path),
f"--comm={comm_addr}",
f"--logs={logs_addr}",
]
```
We should make the config more customizable and composable like
`AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST`.
## What
Current format before refactor:
```
[sdk]
queue_to_sdk = {"java-11": "java", "java-12": "java"}
```
After:
```ini
# Rename the `[sdk] queue_to_sdk` as `[sdk] queue_to_coordinator`
[sdk]
queue_to_coordinator = {"legacy-java-queue": "jdk-11", "modern-java-queue":
"java-17"}
# but we introduce a new `[sdk] coordinators` config
[sdk]
coordinators = [
{
"name": "jdk-11",
"classpath":
"airflow.providers.sdk.java.coordinator.JavaCoordinator",
"kwargs": {
"java_executable": "/usr/lib/jvm/java-11-openjdk-amd64/bin/java",
"jvm_args": ["-Xmx512m"],
"jdk_home": "/usr/lib/jvm/java-11-openjdk-amd64"
}
},
{
"name": "jdk-17",
"classpath":
"airflow.providers.sdk.java.coordinator.JavaCoordinator",
"kwargs": {
"java_executable": "/usr/lib/jvm/java-17-openjdk-amd64/bin/java",
"jvm_args": ["-Xmx1024m", "-Xms256m"],
"jdk_home": "/usr/lib/jvm/java-17-openjdk-amd64"
}
}
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
