Good morning Max and thanks for clarifying!
I generated the JAR 2.19.0 in the second test via the default demo code
from Beam. There were no further adjustments from my side, but as I can see
there are some open points in JIRA for 1.9.2, so for now I think that we
can focus on 1.9.1 as a target.
To understand the fail-safety paradigm correctly - It is my understanding
that, when I hit stop my pipeline (reading from Kafka, writing to BigQuery)
will stop consuming from Kafka and wait until all bundles have been written
out to BigQuery. Meaning that it will also correctly persist the offset of
the message it has successfully read in Kafka. When it gets started again
it will resume at that offset then.
In contrast, hitting cancel causes my pipeline to drop everything it is
doing immediately and leaving me (without taking a snapshot) in an unclear
state.
For operations it is useful to have a cancel button, to be able to quickly
shutdown a pipeline and completely re-read it from a streaming source. For
example when there was a critical bug in the pipeline processing code, or
when a team, that was sending data through that source happened to release
a feature earlier than announced and some data had been missed, as the
schema had not been updated in the pipeline before.
On the other hand, it is useful to have a stop button, to be able to
cleanly shutdown a pipeline when a feature is going to be released
tomorrow, and the pipeline processing code should be updated before,
without the overhead of re-reading the complete past. Taking a snapshot
uses the current representation of the Beam code as a Flink job, but what
if one wants to update the Beam code and thus the pipeline code without the
need to reprocess the whole history? Moreover, a stop button is very useful
when a new Flink version is going to be rolled out: then one can drain all
pipelines (for us there are right now 31), rollout a new Flink version and
start them at the point where they left of with their last committed offset
in Kafka.
Does that make sense?
Best,
Tobi
On Sun, Mar 1, 2020 at 5:23 PM Maximilian Michels <m...@apache.org> wrote:
> In some sense, stop is different because sources will be stopped first
> and then all the downstream operators will stop automatically. However,
> in terms of correctness of your program using cancel or stop does not
> make a difference because neither approach takes a checkpoint.
>
> Only at the time of a checkpoint you are guranteed to have a consistent
> state. A checkpoint is the only way to resume a canceled/stopped
> pipeline correctly.
>
> Does that make sense? I understand that stop may have been convenient
> for your use case. If so, we may consider adding it again.
>
> Cheers,
> Max
>
> PS: Concerning the web interface with 1.9.2, I'm not sure what changes
> your Jar contain but we'll have to look into this when we upgrade to
> 1.9.2 in Beam.
>
> On 28.02.20 14:59, Kaymak, Tobias wrote:
> > I investigated further:
> >
> > As Flink 1.9.1 works and Flink 1.9.2 does not, I simply tried a rollout
> > of a vanilla flink:1.9.2-scala_2.11 image to K8s and that worked. So the
> > issue must be in my image or the JAR I am attaching:
> >
> > ARG FLINK_VERSION=1.9.2
> > ARG SCALA_VERSION=2.11
> > FROM flink:${FLINK_VERSION}-scala_${SCALA_VERSION}
> >
> > COPY --chown=flink:flink conf/log4j-console.properties
> > /opt/flink/conf/log4j-console.properties
> > ADD --chown=flink:flink
> >
> https://repo1.maven.org/maven2/org/apache/flink/flink-metrics-prometheus_${SCALA_VERSION}/${FLINK_VERSION}/flink-metrics-prometheus_${SCALA_VERSION}-${FLINK_VERSION}.jar
> >
> /opt/flink/lib/flink-metrics-prometheus_${SCALA_VERSION}-${FLINK_VERSION}.jar
> > ADD --chown=flink:flink
> >
> https://repo1.maven.org/maven2/org/apache/flink/flink-statebackend-rocksdb_${SCALA_VERSION}/${FLINK_VERSION}/flink-statebackend-rocksdb_${SCALA_VERSION}-${FLINK_VERSION}.jar
> >
> /opt/flink/lib/flink-statebackend-rocksdb_${SCALA_VERSION}-${FLINK_VERSION}.jar
> > ADD --chown=flink:flink deployment/run.sh /opt/flink/run.sh
> > RUN chmod +x /opt/flink/run.sh
> >
> > COPY --from=builder --chown=flink:flink
> > /build/target/di-beam-bundled.jar /opt/flink/lib/beam_pipelines.jar
> >
> > Commenting out the last COPY step - to circumvent the addition of the
> > fat Beam JAR - did the trick. So I think my beam JAR contains something
> > Flink does not like. Next thing I tried was building a vanilla Beam JAR
> via:
> >
> > mvn archetype:generate \
> > -DarchetypeGroupId=org.apache.beam \
> > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > -DarchetypeVersion=2.19.0 \
> > -DgroupId=org.example \
> > -DartifactId=word-count-beam \
> > -Dversion="0.1" \
> > -Dpackage=org.apache.beam.examples \
> > -DinteractiveMode=false
> >
> > and then mvn package -Pflink-runner - even with this JAR the
> > webinterface of the jobmanager does not come up.
> >
> > Best,
> > Tobias
> >
> > On Fri, Feb 28, 2020 at 11:36 AM Kaymak, Tobias
> > <tobias.kay...@ricardo.ch <mailto:tobias.kay...@ricardo.ch>> wrote:
> >
> > Hi Max,
> >
> > I am surprised that Stop and Cancel had the same action in Flink 1.8
> > and lower. When I clicked on stop in the past the whole pipeline
> > shutdown each operator one after the other and after a couple of
> > minutes it was done and all outstanding bundles were imported
> > correctly (my pipeline goes from Kafka to BigQuery). When I clicked
> > cancel it stopped immediately, leaving some data (bundles) in the
> > "air". Was my understanding wrong? From an operational point of view
> > when clicking cancel and starting the pipeline again will it resume
> > without a loss of data?
> >
> > Here is my mail to the Flink mailing list explaining my issue with
> > 1.9.2:
> >
> > I enabled DEBUG/TRACE logging and it looks ok to me when I make the
> > request after the jobmanager pod has been started:
> >
> > 2020-02-27 10:11:42,439 TRACE
> > org.apache.flink.runtime.rest.FileUploadHandler -
> > Received request. URL:/ Method:GET
> > 2020-02-27 10:11:42,440 DEBUG
> >
> org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler
> > - Responding with file
> >
> '/tmp/flink-web-19e3b6a9-f464-49bf-ad94-3237ae898bc7/flink-web-ui/index.html'
> > 2020-02-27 10:11:42,578 TRACE
> > org.apache.flink.runtime.rest.FileUploadHandler -
> > Received request. URL:/styles.30d0912c1ece284d8d9a.css Method:GET
> > 2020-02-27 10:11:42,579 DEBUG
> >
> org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler
> > - Responding with file
> >
>
> '/tmp/flink-web-19e3b6a9-f464-49bf-ad94-3237ae898bc7/flink-web-ui/styles.30d0912c1ece284d8d9a.css'
> > 2020-02-27 10:11:42,648 TRACE
> > org.apache.flink.runtime.rest.FileUploadHandler -
> > Received request. URL:/polyfills.b37850e8279bc3caafc9.js Method:GET
> > 2020-02-27 10:11:42,649 DEBUG
> >
> org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler
> > - Responding with file
> >
>
> '/tmp/flink-web-19e3b6a9-f464-49bf-ad94-3237ae898bc7/flink-web-ui/polyfills.b37850e8279bc3caafc9.js'
> > 2020-02-27 10:11:42,651 TRACE
> > org.apache.flink.runtime.rest.FileUploadHandler -
> > Received request. URL:/runtime.440aa244803781d5f83e.js Method:GET
> > 2020-02-27 10:11:42,652 DEBUG
> >
> org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler
> > - Responding with file
> >
>
> '/tmp/flink-web-19e3b6a9-f464-49bf-ad94-3237ae898bc7/flink-web-ui/runtime.440aa244803781d5f83e.js'
> > 2020-02-27 10:11:42,658 TRACE
> > org.apache.flink.runtime.rest.FileUploadHandler -
> > Received request. URL:/main.177039bdbab11da4f8ac.js Method:GET
> > 2020-02-27 10:11:42,658 DEBUG
> >
> org.apache.flink.runtime.rest.handler.legacy.files.StaticFileServerHandler
> > - Responding with file
> >
>
> '/tmp/flink-web-19e3b6a9-f464-49bf-ad94-3237ae898bc7/flink-web-ui/main.177039bdbab11da4f8ac.js'
> > 2020-02-27 10:11:42,922 TRACE
> > org.apache.flink.runtime.rest.FileUploadHandler -
> > Received request. URL:/config Method:GET
> >
> > I tried Chrome (newest version) and Safari, both with Incognito
> > modes as well. The webserver returns Javascript and HTML to the
> > browser - so I am very puzzled about what could be wrong here?
> >
> > image.png
> >
> > Content type looks ok:
> >
> > curl -i localhost:8081
> > HTTP/1.1 200 OK
> > Content-Type: text/html
> > Date: Thu, 27 Feb 2020 12:48:08 GMT
> > Expires: Thu, 27 Feb 2020 12:53:08 GMT
> > Cache-Control: private, max-age=300
> > Last-Modified: Thu, 27 Feb 2020 12:43:36 GMT
> > Connection: keep-alive
> > content-length: 2137
> >
> >
> > Best,
> > Tobias
> >
> > On Fri, Feb 28, 2020 at 10:29 AM Maximilian Michels <m...@apache.org
> > <mailto:m...@apache.org>> wrote:
> >
> > The stop functionality has been removed in Beam. It was
> > semantically
> > identical to using cancel, so we decided to drop support for it.
> >
> > > Both Flink 1.9.2 and 1.10.0 are not supported yet on Beam,
> > probably they will be part of the 2.21.0 release
> >
> > 1.9.2 should be supported, as it is just a patch release.
> > Generally, all
> > 1.x.y release are compatible with Beam's Flink 1.x Runner.
> >
> > > I can confirm that the pipeline behaves as expected with
> > 2.20.0-SNAPSHOT and Flink 1.9.1 - I also tried Flink 1.9.2 but
> > the webinterface didn't show up (just a blank page - javascript
> > was being loaded though).
> >
> > I'm surprised about that. If you have more information that
> > would be great.
> >
> > -Max
> >
> > On 28.02.20 07:46, Kaymak, Tobias wrote:
> > > What I found so far is that the "Stop" Button next to the
> > "Cancel"
> > > button is missing when I run my Beam 2.19.0/2.20.0-SNAPSHOT
> > streaming
> > > pipeline in Flink's 1.9.1's web interface. I couldn't figure
> > out yet if
> > > it has been removed by the Flink team on purpose or if that
> > is something
> > > "missing" in the Beam translation layer.
> > >
> > > Best,
> > > Tobias
> > >
> > > On Thu, Feb 27, 2020 at 1:44 PM Ismaël Mejía
> > <ieme...@gmail.com <mailto:ieme...@gmail.com>
> > > <mailto:ieme...@gmail.com <mailto:ieme...@gmail.com>>> wrote:
> > >
> > > Both Flink 1.9.2 and 1.10.0 are not supported yet on
> > Beam, probably
> > > they will be part of the 2.21.0 release
> > > You can follow the progress on both issues (and help us
> > with early
> > > testing once in master):
> > >
> > > BEAM-9295 Add Flink 1.10 build target and Make FlinkRunner
> > > compatible with Flink 1.10
> > > https://issues.apache.org/jira/browse/BEAM-9295
> > >
> > > BEAM-9299 Upgrade Flink Runner to 1.8.3 and 1.9.2
> > > https://issues.apache.org/jira/browse/BEAM-9299
> > >
> > > Regards,
> > > Ismaël
> > >
> > >
> > > On Thu, Feb 27, 2020 at 11:53 AM Kaymak, Tobias
> > > <tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>> wrote:
> > >
> > > Copy paste error, sorry:
> > >
> > > 2.20.0-SNAPSHOT in combination
> > with beam-runners-flink-1.10
> > > or beam-runners-flink-1.10-SNAPSHOT didn't work
> > either for me.
> > >
> > >
> > > On Thu, Feb 27, 2020 at 11:50 AM Kaymak, Tobias
> > > <tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>> wrote:
> > >
> > > I can confirm that the pipeline behaves as
> > expected with
> > > 2.20.0-SNAPSHOT and Flink 1.9.1 - I also tried
> > Flink 1.9.2
> > > but the webinterface didn't show up (just a blank
> > page -
> > > javascript was being loaded though).
> > > I emptied my cache and investigated the log and
> > asked on the
> > > Flink mailing list if this is known - maybe it's
> also
> > > because of one of the dependencies in my fat Beam
> > jar. I am
> > > still investigating this.
> > >
> > > How can I test the Flink 1.10 runners? (The
> > following POM is
> > > not resolvable by maven)
> > >
> > > <dependency>
> > > <groupId>org.apache.beam</groupId>
> > >
> > <artifactId>beam-runners-flink-1.10</artifactId>
> > > <version>2.20-SNAPSHOT</version>
> > > </dependency>
> > >
> > > Best,
> > > Tobi
> > >
> > > On Wed, Feb 26, 2020 at 5:07 PM Ismaël Mejía
> > > <ieme...@gmail.com <mailto:ieme...@gmail.com>
> > <mailto:ieme...@gmail.com <mailto:ieme...@gmail.com>>> wrote:
> > >
> > > Since it was merged yesterday you can test
> > with the
> > > 2.20.0-SNAPSHOT until the first candidate is
> out.
> > >
> > > On Wed, Feb 26, 2020 at 4:37 PM Kaymak, Tobias
> > > <tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>
> > > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>> wrote:
> > >
> > > If I am not running in detached mode (so
> > that my
> > > pipeline starts) I am unable to Stop it
> > in the
> > > webinterface. The only option available
> > is to cancel
> > > it. Is this expected?
> > >
> > > Screenshot 2020-02-26 at 16.34.08.png
> > >
> > > On Wed, Feb 26, 2020 at 4:16 PM Kaymak,
> > Tobias
> > > <tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>
> > > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>> wrote:
> > >
> > > Hello,
> > >
> > > we fixed the issue and are ready to
> > test :) - is
> > > there a RC already available?
> > >
> > > Best,
> > > Tobi
> > >
> > > On Wed, Feb 26, 2020 at 12:59 PM
> > Kaymak, Tobias
> > > <tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>
> > > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>> wrote:
> > >
> > > Hello,
> > >
> > > happy to help testing! I am
> > currently fixing
> > > a networking issue between our
> > dev cluster
> > > for integration tests and the
> > Kafka it is
> > > consuming from.
> > > After that I would be ready to
> > spin it up
> > > and test
> > >
> > > Best,
> > > Tobi
> > >
> > > On Mon, Feb 24, 2020 at 10:13 PM
> > Maximilian
> > > Michels <m...@apache.org
> > <mailto:m...@apache.org>
> > > <mailto:m...@apache.org
> > <mailto:m...@apache.org>>> wrote:
> > >
> > > Thank you for reporting /
> > filing /
> > > collecting the issues.
> > >
> > > There is a fix pending:
> > > https://github.com/apache/beam/pull/10950
> > >
> > > As for the upgrade issues,
> > the 1.8 and
> > > 1.9 upgrade is trivial. I will
> > > check out the Flink 1.10 PR
> > tomorrow.
> > >
> > > Cheers,
> > > Max
> > >
> > > On 24.02.20 09:26, Ismaël
> > Mejía wrote:
> > > > We are cutting the release
> > branch for
> > > 2.20.0 next wednesday, so not
> > sure
> > > > if these tickets will make
> > it, but
> > > hopefully.
> > > >
> > > > For ref,
> > > > BEAM-9295 Add Flink 1.10
> > build target
> > > and Make FlinkRunner
> compatible
> > > > with Flink 1.10
> > > > BEAM-9299 Upgrade Flink
> > Runner to
> > > 1.8.3 and 1.9.2
> > > >
> > > > In any case if you have
> > cycles to
> > > help test any of the related
> > tickets
> > > > PRs that would help too.
> > > >
> > > >
> > > > On Mon, Feb 24, 2020 at
> > 8:47 AM
> > > Kaymak, Tobias
> > <tobias.kay...@ricardo.ch <mailto:tobias.kay...@ricardo.ch>
> > >
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>
> > > >
> > <mailto:tobias.kay...@ricardo.ch <mailto:
> tobias.kay...@ricardo.ch>
> > >
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>>> wrote:
> > > >
> > > > Hi Kyle,
> > > >
> > > > thank you for creating
> > the JIRA
> > > ticket, I think my best
> > option right
> > > > now is to wait for a
> > Beam version
> > > that is running on Flink 1.10
> > then
> > > > - unless there is a
> > new Beam
> > > release around the corner :)
> > > >
> > > > Best,
> > > > Tobi
> > > >
> > > > On Thu, Feb 20, 2020
> > at 11:52 PM
> > > Kyle Weaver
> > <kcwea...@google.com <mailto:kcwea...@google.com>
> > > <mailto:kcwea...@google.com
> > <mailto:kcwea...@google.com>>
> > > >
> > <mailto:kcwea...@google.com <mailto:kcwea...@google.com>
> > > <mailto:kcwea...@google.com
> > <mailto:kcwea...@google.com>>>> wrote:
> > > >
> > > > Hi Tobi,
> > > >
> > > > This seems like a
> bug
> > > with Beam 2.19. I filed
> > > >
> > > https://issues.apache.org/jira/browse/BEAM-9345 to
> > > track the issue.
> > > >
> > > > > What puzzles me
> > is that
> > > the session cluster should be
> > allowed
> > > > to have multiple
> > environments
> > > in detached mode - or am I
> wrong?
> > > >
> > > > It looks like
> > that check is
> > > removed in Flink 1.10:
> > > >
> > > https://issues.apache.org/jira/browse/FLINK-15201
> > > >
> > > > Thanks for
> reporting.
> > > > Kyle
> > > >
> > > > On Thu, Feb 20,
> > 2020 at 4:10
> > > AM Kaymak, Tobias
> > > >
> > <tobias.kay...@ricardo.ch <mailto:tobias.kay...@ricardo.ch>
> > >
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>
> > >
> > <mailto:tobias.kay...@ricardo.ch <mailto:
> tobias.kay...@ricardo.ch>
> > >
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>>> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am trying to
> > upgrade
> > > from a Flink session cluster
> > 1.8 to
> > > > 1.9 and from
> > Beam 2.16.0
> > > to 2.19.0.
> > > > Everything went
> > > quite smoothly, the local
> > runner and the
> > > > local Flink
> > runner work
> > > flawlessly.
> > > >
> > > > However when I:
> > > > 1. Generate
> > a Beam jar
> > > for the FlinkRunner via maven
> > (mvn
> > > > package
> > -PFlinkRunner)
> > > > 2. Glue
> > that into a
> > > Flink 1.9 docker image
> > > > 3. Start
> > the image as
> > > a Standalone Session Cluster
> > > >
> > > > When I try to
> > launch the
> > > first pipeline I get the
> > following
> > > > exception
> > > >
> > > >
> > >
> > org.apache.flink.client.program.ProgramInvocationException:
> > > > The main
> > method caused an
> > > error: Failed to construct
> > > > instance from
> > factory method
> > > >
> > >
> > FlinkRunner#fromOptions(interface
> > > >
> > >
> > org.apache.beam.sdk.options.PipelineOptions)
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> > > > Caused by:
> > > java.lang.RuntimeException:
> > Failed to
> > > construct
> > > > instance from
> > factory method
> > > >
> > >
> > FlinkRunner#fromOptions(interface
> > > >
> > >
> > org.apache.beam.sdk.options.PipelineOptions)
> > > > at
> > > >
> > >
> >
>
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
> > > > at
> > > >
> > >
> >
> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
> > > > at
> > > >
> > >
> >
> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
> > > > at
> > >
> > org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
> > > > at
> > >
> > org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
> > > > at
> > > >
> > >
> >
> ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180)
> > > > at
> > > >
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > >
> > >
> >
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at
> > >
> > java.lang.reflect.Method.invoke(Method.java:498)
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
> > > > ... 9
> > more
> > > > Caused by:
> > >
> > java.lang.reflect.InvocationTargetException
> > > > at
> > > >
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > >
> > >
> >
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at
> > >
> > java.lang.reflect.Method.invoke(Method.java:498)
> > > > at
> > > >
> > >
> >
>
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
> > > > ...
> > 19 more
> > > > Caused by:
> > > >
> > >
> > org.apache.flink.api.common.InvalidProgramException:
> > > > Multiple
> > environments
> > > cannot be created in detached
> > mode
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67)
> > > > at
> > >
> > java.util.Optional.map(Optional.java:215)
> > > > at
> > > >
> > >
> >
>
> org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068)
> > > > at
> > > >
> > >
> >
>
> org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43)
> > > > at
> > > >
> > >
> >
> org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96)
> > > > at
> > > >
> > >
> >
> org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90)
> > > > ...
> > 24 more
> > > >
> > > > I've checked
> > the release
> > > notes and the issues and
> couldn't
> > > > find anything
> that
> > > relates to this. What puzzles
> > me is that
> > > > the session
> > cluster
> > > should be allowed to have
> > multiple
> > > > environments
> > in detached
> > > mode - or am I wrong?
> > > >
> > > > Best,
> > > > Tobi
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Tobias Kaymak
> > > > Data Engineer
> > > > Data Intelligence
> > > >
> > > > tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>
> > >
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>
> > >
> > <mailto:tobias.kay...@ricardo.ch <mailto:
> tobias.kay...@ricardo.ch>
> > >
> > <mailto:tobias.kay...@ricardo.ch
> > <mailto:tobias.kay...@ricardo.ch>>>
> > > > www.ricardo.ch
> > <http://www.ricardo.ch>
> > > <http://www.ricardo.ch>
> > > <http://www.ricardo.ch/>
> > > > Theilerstrasse 1a,
> > 6300 Zug
> > > >
> > >
> >
>
--
Tobias Kaymak
Data Engineer
Data Intelligence
tobias.kay...@ricardo.ch
www.ricardo.ch
Theilerstrasse 1a, 6300 Zug
