[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721719#comment-16721719 ] Giorgos Stamatakis commented on BEAM-5419: -- Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java -Dexec.mainClass=myPipeline "-Dexec.args=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[
https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721392#comment-16721392
]
Maximilian Michels commented on BEAM-5419:
--
No worries about asking questions! Thanks for trying out the 1.6 build. If you
just replace the dependency in the pom file, e.g. {{beam-runners-flink_2.11}}
against {{beam-runners-flink-1.6}}, it will only work if you installed the 1.6
version in your local Maven repository.
Like so: {{mvn install:install-file -Dfile=}}
> Build multiple versions of the Flink Runner against different Flink versions
>
>
> Key: BEAM-5419
> URL: https://issues.apache.org/jira/browse/BEAM-5419
> Project: Beam
> Issue Type: New Feature
> Components: build-system, runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.10.0
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Following up on a discussion on the mailing list.
> We want to keep the Flink version stable across different versions to avoid
> upgrade pain for long-term users. At the same time, there are users out there
> with newer Flink clusters and developers also want to utilize new Flink
> features.
> It would be great to build multiple versions of the Flink Runner against
> different Flink versions.
> When the upgrade is as simple as changing the version property in the build
> script, this should be pretty straight-forward. If not, having a "base
> version" and applying a patch during the build could be an option. We should
> avoid duplicating any Runner code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721288#comment-16721288 ] Giorgos Stamatakis commented on BEAM-5419: -- This is my last question since I have already asked a lot of them. Im trying to replace in pom.xml the already existing beam-runners-flink_2.11 dependency (which works up to 1.5.5) with the new one from the compiled sources beam-runners-flink-1.6-2.10.0-SNAPSHOT.jar. But its still undetected when Im running the pipeline (error: cant find FlinkRunner) > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721244#comment-16721244 ] Maximilian Michels commented on BEAM-5419: -- The error message implies that the jar has not been correctly added to the compile path. Or the jar does not contain the FlinkRunner classes, but I've just checked on my machine that this is the case. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721112#comment-16721112 ] Giorgos Stamatakis commented on BEAM-5419: -- I got the jars by building from source with gradle. I used the jars that were in the libs folder of 1.6. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721110#comment-16721110 ] Maximilian Michels commented on BEAM-5419: -- This just has been merged to master. Unfortunately, you will have to wait for the next release to be able to retrieve the artifacts via Maven. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720632#comment-16720632 ] Giorgos Stamatakis commented on BEAM-5419: -- Quick question. I tried to build my project with mvn with the 1.6 jar and it fails with "Flink runner not found". Still runs ok with 1.5.5 though. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719425#comment-16719425 ] Maximilian Michels commented on BEAM-5419: -- [~djharper] Thanks for the pointer to the AWS service. Indeed, it is plain Flink 1.6, so you would be able to run Beam on it as well. The PR was merged for supporting multiple versions of the Flink Runner. So the next release will contain Flink 1.6 and Flink 1.7. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714705#comment-16714705 ] Daniel Harper commented on BEAM-5419: - Additionally AWS have recently released a new managed flink product [1] that's running 1.6. I'd be eager to try running a job that's using the BEAM API (theoretically it should work?) but we need the flink runner to support 1.6 first. [1] https://aws.amazon.com/kinesis/data-analytics/ > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714699#comment-16714699 ] Daniel Harper commented on BEAM-5419: - This would be of great interest to us too, we run on AWS EMR which the latest version includes flink 1.6.x, but we can't upgrade until BEAM supports this version. AWS release train looks to be about once a month for EMR, so I'd imagine they'll roll out Flink 1.7 in the next few months as an option. Right now tracking the releases/compatibility of BEAM + Flink + EMR is becoming quite an effort > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710251#comment-16710251 ] Jeroen Steggink commented on BEAM-5419: --- [~mxm], that's great! I'm interested in 1.6.x and 1.7.x. Different projects, with different needs. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710157#comment-16710157 ] Maximilian Michels commented on BEAM-5419: -- Since this gets asked for more and more, I will prioritize this for 2.10.0. With Flink 1.7 out, we should provide at least builds for 1.7 and 1.6, and most likely also 1.5 because supporting it is about the same effort as supporting 1.6. [~jeroens] Are you interested in 1.6 or 1.7? > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708998#comment-16708998 ] Jeroen Steggink commented on BEAM-5419: --- In the meantime Flink 1.7 was released last week. Is anyone working on different Flink runner builds? > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702408#comment-16702408 ] Maximilian Michels commented on BEAM-5419: -- [~gstamatakis] The above error is actually just a renamed class linked in a JavaDoc comment. It should compile fine if you remove the mentioned line (it is just a comment). > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699201#comment-16699201 ] Giorgos Stamatakis commented on BEAM-5419: -- Thank you for the reply. I used 2.8.0,2.9.0 and the master branch. Here are the ./gradlew build --stacktrace results after changing the runners/flink/build.gradle flink version line to 1.6.0 (same for 1.6.1). > Task :beam-runners-flink_2.11:compileJava FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-runners-flink_2.11:compileJava'. > Compilation failed with exit code 1; see the compiler error output for > details. * Try: Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':beam-runners-flink_2.11:compileJava'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77) at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51) at org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105) at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54) at org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:79) at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101) at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44) at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91) at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62) at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54) at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43) at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292) at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174) at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90) at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46) at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130) at
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699167#comment-16699167 ] Maximilian Michels commented on BEAM-5419: -- [~gstamatakis] The last time I tried it was only a minor issues with classes being renamed. Maybe post the error message? > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698115#comment-16698115 ] Giorgos Stamatakis commented on BEAM-5419: -- I did try changing the version from 1.5.5 to 1.6.0 (and 1.6.1) but the beam-runner-flink still fails. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697920#comment-16697920 ] Thomas Weise commented on BEAM-5419: Mailing list thread: [https://lists.apache.org/thread.html/e44d07f4ed12eda9af38fb263a1f53e6e82e838cce1fbb82fe51c6df@%3Cdev.beam.apache.org%3E] To repeat from there for watchers: You can build Beam yourself for Flink 1.6 by changing the version here: https://github.com/apache/beam/blob/v2.8.0/runners/flink/build.gradle#L49 > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697919#comment-16697919 ] Thomas Weise commented on BEAM-5419: We should provide binaries for 1.6.x for the next release. I see that EMR now supports 1.6.1: [https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-flink.html] - Bigtop on the other hand is still at 1.4x: [https://github.com/apache/bigtop/blob/master/bigtop.bom#L309] > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693634#comment-16693634 ] Maximilian Michels commented on BEAM-5419: -- With the release cut coming up tomorrow, likely not. I started looking into this but there were some difficulties with regard to the overall build system. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.9.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690526#comment-16690526 ] Giorgos Stamatakis commented on BEAM-5419: -- Will this be fixed in 2.9.0? > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.9.0 > > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
