azagrebin commented on a change in pull request #11427: [FLINK-15790][k8s] Make
FlinkKubeClient and its implementations asynchronous
URL: https://github.com/apache/flink/pull/11427#discussion_r406097093
##########
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
##########
@@ -61,158 +61,180 @@
private final String clusterId;
private final String nameSpace;
- public Fabric8FlinkKubeClient(Configuration flinkConfig,
KubernetesClient client) {
+ private final ExecutorWrapper executorWrapper;
+
+ public Fabric8FlinkKubeClient(Configuration flinkConfig,
KubernetesClient client, ExecutorWrapper executorWrapper) {
this.flinkConfig = checkNotNull(flinkConfig);
this.internalClient = checkNotNull(client);
this.clusterId =
checkNotNull(flinkConfig.getString(KubernetesConfigOptions.CLUSTER_ID));
this.nameSpace =
flinkConfig.getString(KubernetesConfigOptions.NAMESPACE);
+
+ this.executorWrapper = executorWrapper;
}
@Override
- public void createJobManagerComponent(KubernetesJobManagerSpecification
kubernetesJMSpec) {
+ public CompletableFuture<Void>
createJobManagerComponent(KubernetesJobManagerSpecification kubernetesJMSpec) {
final Deployment deployment = kubernetesJMSpec.getDeployment();
final List<HasMetadata> accompanyingResources =
kubernetesJMSpec.getAccompanyingResources();
// create Deployment
LOG.debug("Start to create deployment with spec {}",
deployment.getSpec().toString());
- final Deployment createdDeployment = this.internalClient
- .apps()
- .deployments()
- .inNamespace(this.nameSpace)
- .create(deployment);
-
- // Note that we should use the uid of the created Deployment
for the OwnerReference.
- setOwnerReference(createdDeployment, accompanyingResources);
- this.internalClient
- .resourceList(accompanyingResources)
- .inNamespace(this.nameSpace)
- .createOrReplace();
+ return CompletableFuture.runAsync(() -> {
+ final Deployment createdDeployment = this.internalClient
+ .apps()
+ .deployments()
+ .inNamespace(this.nameSpace)
+ .create(deployment);
+
+ // Note that we should use the uid of the created
Deployment for the OwnerReference.
+ setOwnerReference(createdDeployment,
accompanyingResources);
+
+ this.internalClient
+ .resourceList(accompanyingResources)
+ .inNamespace(this.nameSpace)
+ .createOrReplace();
+ }, executorWrapper.getExecutor());
}
@Override
public void createTaskManagerPod(KubernetesPod kubernetesPod) {
- final Deployment masterDeployment = this.internalClient
- .apps()
- .deployments()
- .inNamespace(this.nameSpace)
- .withName(KubernetesUtils.getDeploymentName(clusterId))
- .get();
-
- if (masterDeployment == null) {
- throw new RuntimeException(
- "Failed to find Deployment named " + clusterId
+ " in namespace " + this.nameSpace);
- }
+ CompletableFuture.runAsync(() -> {
Review comment:
I checked the JM implementation and it looks that JM decides when to timeout
the pending request. Judging the existing active RM integrations (e.g. yarn),
the RM should retry slot allocation until JM calls `RM#cancelSlotRequest` which
happens in case of JM slot request timeout. Therefore, I still think that
following `KubernetesResourceManager#onError` is more correct if internal
fabric8 http request fails (still needs a unit test if not added yet).
My concern is more about a back-off strategy that the
`KubernetesResourceManager` does not retry
`FlinkKubeClient#createTaskManagerPod` too often but there is a certain time
interval between retries. Maybe, something in fabric8 `Config` (like
`KUBERNETES_REQUEST_TIMEOUT_SYSTEM_PROPERTY`) already can handle this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
