[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException
[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586248#comment-15586248 ] Jason Kuster commented on BEAM-769: --- This also seems to have caused a postcommit failure: https://builds.apache.org/job/beam_PostCommit_MavenVerify/1567/ > Spark translation tests failing with NullPointerException > - > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: Not applicable >Reporter: Daniel Halperin >Assignee: Amit Sela > > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/ > {code} > org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > Amit, can you please take a look? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException
[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586335#comment-15586335 ] Amit Sela commented on BEAM-769: >From my experience those failures happen when a "bad" context is reused - >since the context is reused in Spark tests, if it failed and closed for some >reason, it will try to keep reusing that closed context, and that's how it >manifests. I fixed this while working on BEAM-259, this is the fix: https://github.com/apache/incubator-beam/pull/1055/files#diff-cdc52a99e44ad1b6d1950ecab0928ecd. I could patch this up as part of this ticket, because BEAM-259 is still in review, but it will be tomorrow morning (IDT). > Spark translation tests failing with NullPointerException > - > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: Not applicable >Reporter: Daniel Halperin >Assignee: Amit Sela > > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/ > {code} > org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > Amit, can you please take a look? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException
[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588301#comment-15588301 ] Amit Sela commented on BEAM-769: [~kenn] do you think PR#1055 will be ready for merge soon ? I would generally fix such issue immediately, but it's flakey, only happens if something failed (unexpectedly) and will cause a "domino effect" on following tests, so I might wait for PR#1055 if we can merge it soon. > Spark translation tests failing with NullPointerException > - > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: Not applicable >Reporter: Daniel Halperin >Assignee: Amit Sela > > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/ > {code} > org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > Amit, can you please take a look? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException
[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590064#comment-15590064 ] Amit Sela commented on BEAM-769: Merging BEAM-259 which should resolve this, I'll wait a day or two to see it does and then close this ticket as duplicate. > Spark translation tests failing with NullPointerException > - > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: Not applicable >Reporter: Daniel Halperin >Assignee: Amit Sela > > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/ > {code} > org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > Amit, can you please take a look? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException
[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592641#comment-15592641 ] Daniel Halperin commented on BEAM-769: -- Looks like it has not been resolved yet. There are 3 more flakes today that include the fix referenced in BEAM-259. https://builds.apache.org/job/beam_PostCommit_MavenVerify/1586/ https://builds.apache.org/job/beam_PostCommit_MavenVerify/1587/ https://builds.apache.org/job/beam_PostCommit_MavenVerify/1588/ Can you take another look? > Spark translation tests failing with NullPointerException > - > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: Not applicable >Reporter: Daniel Halperin >Assignee: Amit Sela > > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/ > {code} > org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > Amit, can you please take a look? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException
[ https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592682#comment-15592682 ] Amit Sela commented on BEAM-769: Yeah I've noticed, if you look closely you'll see that it's not NPE anymore. The problem seems to be Jenkins being horribly slow; Spark streaming tests have a timeout, and if nothing was processed until a hard timeout was reached, it'll rightfully fail. I'll change the ticket accordingly. I can set a graceful stop - meaning that the test will finish any currently processing batches - but that'll make those tests run X2 or even X3 longer most of the time. Thoughts ? > Spark translation tests failing with NullPointerException > - > > Key: BEAM-769 > URL: https://issues.apache.org/jira/browse/BEAM-769 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: Not applicable >Reporter: Daniel Halperin >Assignee: Amit Sela > > https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/ > {code} > org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows > org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded > org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun > org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows > {code} > Amit, can you please take a look? -- This message was sent by Atlassian JIRA (v6.3.4#6332)