[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721719#comment-16721719 ] Giorgos Stamatakis edited comment on BEAM-5419 at 12/14/18 8:56 PM: Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/CheckpointingMode mvn package exec:java Dexec.mainClass=myPipeline "Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime If there is a sample pom.xml Im interested was (Author: gstamatakis): Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/CheckpointingMode mvn package exec:java Dexec.mainClass=myPipeline "Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721719#comment-16721719 ] Giorgos Stamatakis edited comment on BEAM-5419 at 12/14/18 7:25 PM: Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java Dexec.mainClass=myPipeline "Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime was (Author: gstamatakis): Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java Dexec.mainClass=myPipeline "-Dexec.args=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721719#comment-16721719 ] Giorgos Stamatakis edited comment on BEAM-5419 at 12/14/18 7:27 PM: Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/CheckpointingMode mvn package exec:java Dexec.mainClass=myPipeline "Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime was (Author: gstamatakis): Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java Dexec.mainClass=myPipeline "Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721719#comment-16721719 ] Giorgos Stamatakis edited comment on BEAM-5419 at 12/14/18 7:24 PM: Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java Dexec.mainClass=myPipeline "-Dexec.args=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime was (Author: gstamatakis): Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java -Dexec.mainClass=myPipeline "-Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721719#comment-16721719 ] Giorgos Stamatakis edited comment on BEAM-5419 at 12/14/18 7:24 PM: Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java -Dexec.mainClass=myPipeline "-Dexec.args-=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime was (Author: gstamatakis): Thank you for your time but it appears that even after installing (and including in the pom.xml) the flink-1.6 ~3MB jar an error still pops up: Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionMode mvn package exec:java -Dexec.mainClass=myPipeline "-Dexec.args=--runner=FlinkRunner --flinkMaster=localhost:8081 --streaming=true --parallelism=4 --windowSize=10 --filesToStage=target/XXX-bundled-flink.jar" -Pflink-runner Trying to run on a 1.6.2 cluster.The pom.xml flink profile is the following: flink-runner org.apache.beam beam-runners-flink-1.6 2.10.0 runtime > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721244#comment-16721244 ] Maximilian Michels edited comment on BEAM-5419 at 12/14/18 10:59 AM: - The error message implies that the jar has not been correctly added to the compile path. Or the jar does not contain the FlinkRunner classes, but I've just checked on my machine that it contains all the required classes. was (Author: mxm): The error message implies that the jar has not been correctly added to the compile path. Or the jar does not contain the FlinkRunner classes, but I've just checked on my machine that this is the case. > Build multiple versions of the Flink Runner against different Flink versions > > > Key: BEAM-5419 > URL: https://issues.apache.org/jira/browse/BEAM-5419 > Project: Beam > Issue Type: New Feature > Components: build-system, runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.10.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Following up on a discussion on the mailing list. > We want to keep the Flink version stable across different versions to avoid > upgrade pain for long-term users. At the same time, there are users out there > with newer Flink clusters and developers also want to utilize new Flink > features. > It would be great to build multiple versions of the Flink Runner against > different Flink versions. > When the upgrade is as simple as changing the version property in the build > script, this should be pretty straight-forward. If not, having a "base > version" and applying a patch during the build could be an option. We should > avoid duplicating any Runner code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5419) Build multiple versions of the Flink Runner against different Flink versions
[ https://issues.apache.org/jira/browse/BEAM-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699201#comment-16699201 ] Giorgos Stamatakis edited comment on BEAM-5419 at 11/26/18 4:14 PM: Thank you for the reply. I used 2.8.0,2.9.0 and the master branch. Here are the ./gradlew build --stacktrace results after changing the runners/flink/build.gradle flink version line to 1.6.0 (same for 1.6.1). > Task :beam-runners-flink_2.11:compileJava FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':beam-runners-flink_2.11:compileJava'. > Compilation failed with exit code 1; see the compiler error output for > details. * Try: Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Exception is: org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':beam-runners-flink_2.11:compileJava'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77) at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51) at org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105) at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54) at org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:79) at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101) at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44) at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91) at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62) at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59) at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54) at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43) at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300) at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292) at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174) at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90) at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46) at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46
