[
https://issues.apache.org/jira/browse/FLINK-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288780#comment-15288780
]
ASF GitHub Bot commented on FLINK-3667:
---------------------------------------
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1978#issuecomment-219992759
Thank you for working on this!
The flink-client is pretty complicated and I like this pull request as it
is reducing the complexity.
Maybe (depending on the implications) we can further simplify the code and
introduce some nice abstractions to make the code more maintainable.
I'll think about it a bit and get back to the PR once all the issues from
the first pass have been addressed.
> Generalize client<->cluster communication
> -----------------------------------------
>
> Key: FLINK-3667
> URL: https://issues.apache.org/jira/browse/FLINK-3667
> Project: Flink
> Issue Type: Improvement
> Components: YARN Client
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
>
> Here are some notes I took when inspecting the client<->cluster classes with
> regard to future integration of other resource management frameworks in
> addition to Yarn (e.g. Mesos).
> {noformat}
> 1 Cluster Client Abstraction
> ════════════════════════════
> 1.1 Status Quo
> ──────────────
> 1.1.1 FlinkYarnClient
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Holds the cluster configuration (Flink-specific and Yarn-specific)
> • Contains the deploy() method to deploy the cluster
> • Creates the Hadoop Yarn client
> • Receives the initial job manager address
> • Bootstraps the FlinkYarnCluster
> 1.1.2 FlinkYarnCluster
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Wrapper around the Hadoop Yarn client
> • Queries cluster for status updates
> • Life time methods to start and shutdown the cluster
> • Flink specific features like shutdown after job completion
> 1.1.3 ApplicationClient
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Acts as a middle-man for asynchronous cluster communication
> • Designed to communicate with Yarn, not used in Standalone mode
> 1.1.4 CliFrontend
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Deeply integrated with FlinkYarnClient and FlinkYarnCluster
> • Constantly distinguishes between Yarn and Standalone mode
> • Would be nice to have a general abstraction in place
> 1.1.5 Client
> ╌╌╌╌╌╌╌╌╌╌╌╌
> • Job submission and Job related actions, agnostic of resource framework
> 1.2 Proposal
> ────────────
> 1.2.1 ClusterConfig (before: AbstractFlinkYarnClient)
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Extensible cluster-agnostic config
> • May be extended by specific cluster, e.g. YarnClusterConfig
> 1.2.2 ClusterClient (before: AbstractFlinkYarnClient)
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Deals with cluster (RM) specific communication
> • Exposes framework agnostic information
> • YarnClusterClient, MesosClusterClient, StandaloneClusterClient
> 1.2.3 FlinkCluster (before: AbstractFlinkYarnCluster)
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Basic interface to communicate with a running cluster
> • Receives the ClusterClient for cluster-specific communication
> • Should not have to care about the specific implementations of the
> client
> 1.2.4 ApplicationClient
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • Can be changed to work cluster-agnostic (first steps already in
> FLINK-3543)
> 1.2.5 CliFrontend
> ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
> • CliFrontend does never have to differentiate between different
> cluster types after it has determined which cluster class to load.
> • Base class handles framework agnostic command line arguments
> • Pluggables for Yarn, Mesos handle specific commands
> {noformat}
> I would like to create/refactor the affected classes to set us up for a more
> flexible client side resource management abstraction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
