[
https://issues.apache.org/jira/browse/FLINK-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880947#comment-16880947
]
Shuyi Chen commented on FLINK-13132:
------------------------------------
Hi [~till.rohrmann], as [~ZhenqiuHuang] mentioned, we have a central deployer
service managing 1000+ production jobs. Normally, when the YARN cluster is
healthy, the deployment request rate to the deployer service is relatively low,
<1qps. However, in case of cluster maintenance or cluster failures, the
deployer service will be overwhelmed with deployment requests for, in worst
case, all 1000+ production jobs in a short period of time.
If we generate the job graph in the client side, which means that job graph
generation happens in the deployer service in our case, we need to download the
job jar to the deployer service in order to generate the job graph. In case of
the entire YARN cluster failure we mentioned above, the deployer service, which
normally have only a few instances, will become bottleneck both in network &
CPU/memory because it needs to download all 1000+ job jars and generate all job
graphs in a short time, which is about 100GB (say 100MB per job jar on
average). And this will violate our SLA, or we have to overprovision the
deployer service by a lot to spread out both the network and the cpu/memory
load. However, if we generate the job graph in the Job Entrypoint, the load
will naturally distribute among all nodes across the YARN cluster (i.e., a few
hundreds worker nodes on tens of racks), and this is more scalable & resource
efficient.
As for the non-deterministic problem, could we generate the job graph once when
the job is first started, and save the generated job graph in zookeeper, and
when master failover happens, we just use the generated job graph in zookeeper
to restart the job? Thanks.
> Allow ClusterEntrypoints use user main method to generate job graph
> -------------------------------------------------------------------
>
> Key: FLINK-13132
> URL: https://issues.apache.org/jira/browse/FLINK-13132
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.8.0, 1.8.1
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Minor
>
> We are building a service that can transparently deploy a job to different
> cluster management systems, such as Yarn and another internal system. It is
> very cost to download the jar and generate JobGraph in the client side. Thus,
> I want to propose an improvement to make Yarn Entrypoints can be configurable
> to use either FileJobGraphRetriever or ClassPathJobGraphRetriever. It is
> actually a long asking TODO in AbstractionYarnClusterDescriptor in line 834.
> https://github.com/apache/flink/blob/21468e0050dc5f97de5cfe39885e0d3fd648e399/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L834
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
