[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[
https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146287&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146287
]
ASF GitHub Bot logged work on BEAM-5444:
Author: ASF GitHub Bot
Created on: 21/Sep/18 08:34
Start Date: 21/Sep/18 08:34
Worklog Time Spent: 10m
Work Description: mxm closed pull request #6449: [BEAM-5444] Revert Flink
PostCommit to parallelism 1
URL: https://github.com/apache/beam/pull/6449
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index 31b5cf8d846..c2e251ff5d9 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -97,7 +97,12 @@ def createValidatesRunnerTask(Map m) {
group = "Verification"
def runnerType = config.streaming ? "streaming" : "batch"
description = "Validates the ${runnerType} runner"
-def pipelineOptions = JsonOutput.toJson(["--runner=TestFlinkRunner",
"--streaming=${config.streaming}"])
+def pipelineOptions = JsonOutput.toJson(
+["--runner=TestFlinkRunner",
+ "--streaming=${config.streaming}",
+ // TODO This should be changed to > 1
+ "--parallelism=1",
+])
systemProperty "beamTestPipelineOptions", pipelineOptions
classpath = configurations.validatesRunner
testClassesDirs =
files(project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 146287)
Time Spent: 1h 10m (was: 1h)
> Post Commit Failure in Flink Runner
> ---
>
> Key: BEAM-5444
> URL: https://issues.apache.org/jira/browse/BEAM-5444
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, test-failures
>Reporter: Robert Burke
>Assignee: Maximilian Michels
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The first instance of this failure included entirely your commits, could you
> investigate?
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/]
> It appears expected exceptions are either being swallowed and not thrown.
> SEVERE: Pipeline execution failed
> org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception:
> The user defined 'open()' method caused an exception: java.lang.Exception:
> Hasn't yet thrown
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146286&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146286 ] ASF GitHub Bot logged work on BEAM-5444: Author: ASF GitHub Bot Created on: 21/Sep/18 08:32 Start Date: 21/Sep/18 08:32 Worklog Time Spent: 10m Work Description: mxm commented on issue #6449: [BEAM-5444] Revert Flink PostCommit to parallelism 1 URL: https://github.com/apache/beam/pull/6449#issuecomment-423457238 Thanks for keeping an eye on the build! >I realized that we can just run the Flink validates runner test to make sure this fixes the issue, rather than speculating. If it passes, LGTM. PostCommit includes Flink Validates Runner, no? Though much quicker to run only the Flink parts. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146286) Time Spent: 1h (was: 50m) > Post Commit Failure in Flink Runner > --- > > Key: BEAM-5444 > URL: https://issues.apache.org/jira/browse/BEAM-5444 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Robert Burke >Assignee: Maximilian Michels >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > The first instance of this failure included entirely your commits, could you > investigate? > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/] > It appears expected exceptions are either being swallowed and not thrown. > SEVERE: Pipeline execution failed > org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: > The user defined 'open()' method caused an exception: java.lang.Exception: > Hasn't yet thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146151&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146151 ] ASF GitHub Bot logged work on BEAM-5444: Author: ASF GitHub Bot Created on: 20/Sep/18 22:24 Start Date: 20/Sep/18 22:24 Worklog Time Spent: 10m Work Description: lostluck commented on issue #6449: [BEAM-5444] Revert Flink PostCommit to parallelism 1 URL: https://github.com/apache/beam/pull/6449#issuecomment-423353838 I realized that we can just *run* the Flink validates runner test to make sure this fixes the issue, rather than speculating. If it passes, LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146151) Time Spent: 50m (was: 40m) > Post Commit Failure in Flink Runner > --- > > Key: BEAM-5444 > URL: https://issues.apache.org/jira/browse/BEAM-5444 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Robert Burke >Assignee: Maximilian Michels >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The first instance of this failure included entirely your commits, could you > investigate? > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/] > It appears expected exceptions are either being swallowed and not thrown. > SEVERE: Pipeline execution failed > org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: > The user defined 'open()' method caused an exception: java.lang.Exception: > Hasn't yet thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146149 ] ASF GitHub Bot logged work on BEAM-5444: Author: ASF GitHub Bot Created on: 20/Sep/18 22:23 Start Date: 20/Sep/18 22:23 Worklog Time Spent: 10m Work Description: lostluck commented on issue #6449: [BEAM-5444] Revert Flink PostCommit to parallelism 1 URL: https://github.com/apache/beam/pull/6449#issuecomment-423353598 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146149) Time Spent: 40m (was: 0.5h) > Post Commit Failure in Flink Runner > --- > > Key: BEAM-5444 > URL: https://issues.apache.org/jira/browse/BEAM-5444 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Robert Burke >Assignee: Maximilian Michels >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > The first instance of this failure included entirely your commits, could you > investigate? > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/] > It appears expected exceptions are either being swallowed and not thrown. > SEVERE: Pipeline execution failed > org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: > The user defined 'open()' method caused an exception: java.lang.Exception: > Hasn't yet thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146147&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146147 ] ASF GitHub Bot logged work on BEAM-5444: Author: ASF GitHub Bot Created on: 20/Sep/18 22:12 Start Date: 20/Sep/18 22:12 Worklog Time Spent: 10m Work Description: lostluck commented on issue #6449: [BEAM-5444] Revert Flink PostCommit to parallelism 1 URL: https://github.com/apache/beam/pull/6449#issuecomment-423350982 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146147) Time Spent: 0.5h (was: 20m) > Post Commit Failure in Flink Runner > --- > > Key: BEAM-5444 > URL: https://issues.apache.org/jira/browse/BEAM-5444 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Robert Burke >Assignee: Maximilian Michels >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The first instance of this failure included entirely your commits, could you > investigate? > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/] > It appears expected exceptions are either being swallowed and not thrown. > SEVERE: Pipeline execution failed > org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: > The user defined 'open()' method caused an exception: java.lang.Exception: > Hasn't yet thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146103 ] ASF GitHub Bot logged work on BEAM-5444: Author: ASF GitHub Bot Created on: 20/Sep/18 19:12 Start Date: 20/Sep/18 19:12 Worklog Time Spent: 10m Work Description: mxm commented on issue #6449: [BEAM-5444] Revert Flink PostCommit to parallelism 1 URL: https://github.com/apache/beam/pull/6449#issuecomment-423299681 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146103) Time Spent: 20m (was: 10m) > Post Commit Failure in Flink Runner > --- > > Key: BEAM-5444 > URL: https://issues.apache.org/jira/browse/BEAM-5444 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Robert Burke >Assignee: Maximilian Michels >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The first instance of this failure included entirely your commits, could you > investigate? > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/] > It appears expected exceptions are either being swallowed and not thrown. > SEVERE: Pipeline execution failed > org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: > The user defined 'open()' method caused an exception: java.lang.Exception: > Hasn't yet thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5444) Post Commit Failure in Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-5444?focusedWorklogId=146102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146102 ] ASF GitHub Bot logged work on BEAM-5444: Author: ASF GitHub Bot Created on: 20/Sep/18 19:12 Start Date: 20/Sep/18 19:12 Worklog Time Spent: 10m Work Description: mxm opened a new pull request #6449: [BEAM-5444] Revert Flink PostCommit to parallelism 1 URL: https://github.com/apache/beam/pull/6449 As of 8f94a03, the parallelism defaults to the number of available CPU cores in local setups. Since this breaks PostCommit, we default to parallelism 1 for those tests. CC @lostluck Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146102) Time Spent: 10m Remaining Estimate: 0h > Post Commit Failure in Flink Runner > --- > > Key: BEAM-5444 > URL: https://issues.apache.org/jira/browse/BEAM-5444 > Project: Beam > Issue Type: Bug > Components: runner-flink, test-failures >Reporter: Robert Burke >Assignee: Maximilian Michels >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The first instance of this failure included entirely your commits, could you > investigate? > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/1544/] > It appears expected exceptions are either being swallowed and not thrown. > SEVERE: Pipeline execution failed > org.apache.flink.runtime.client.JobExecutionException: java.lang.Exception: > The user defined 'open()' method caused an exception: java.lang.Exception: > Hasn't yet thrown -- This message was sent by Atlassian JIRA (v7.6.3#76005)
