[ https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580540#comment-16580540 ]
Thomas Weise commented on BEAM-5110: ------------------------------------ Since the latter may be the desired behavior (no singleton) and unless it is not working for other reasons, I think this is more a matter of documentation than trying to enforce a particular assumption? > Reconile Flink JVM singleton management with deployment > ------------------------------------------------------- > > Key: BEAM-5110 > URL: https://issues.apache.org/jira/browse/BEAM-5110 > Project: Beam > Issue Type: Bug > Components: runner-flink > Reporter: Ben Sidhom > Assignee: Ben Sidhom > Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > [~angoenka] noticed through debugging that multiple instances of > BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when > executing in standalone cluster mode. This context factory is responsible for > maintaining singleton state across a TaskManager (JVM) in order to share SDK > Environments across workers in a given job. The multiple-loading breaks > singleton semantics and results in an indeterminate number of Environments > being created. > It turns out that the [Flink classloading > mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html] > is determined by deployment mode. Note that "user code" as referenced by > this link is actually the Flink job server jar. Actual end-user code lives > inside of the SDK Environment and uploaded artifacts. > In order to maintain singletons without resorting to IPC (for example, using > file locks and/or additional gRPC servers), we need to force non-dynamic > classloading. For example, this happens when jobs are submitted to YARN for > one-off deployments via `flink run`. However, connecting to an existing > (Flink standalone) deployment results in dynamic classloading. > We should investigate this behavior and either document (and attempt to > enforce) deployment modes that are consistent with our requirements, or (if > possible) create a custom classloader that enforces singleton loading. -- This message was sent by Atlassian JIRA (v7.6.3#76005)