[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

2016-10-18 Thread Jason Kuster (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586248#comment-15586248
 ] 

Jason Kuster commented on BEAM-769:
---

This also seems to have caused a postcommit failure:

https://builds.apache.org/job/beam_PostCommit_MavenVerify/1567/

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

2016-10-18 Thread Amit Sela (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586335#comment-15586335
 ] 

Amit Sela commented on BEAM-769:


>From my experience those failures happen when a "bad" context is reused - 
>since the context is reused in Spark tests, if it failed and closed for some 
>reason, it will try to keep reusing that closed context, and that's how it 
>manifests.
I fixed this while working on BEAM-259, this is the fix: 
https://github.com/apache/incubator-beam/pull/1055/files#diff-cdc52a99e44ad1b6d1950ecab0928ecd.
I could patch this up as part of this ticket, because BEAM-259 is still in 
review, but it will be tomorrow morning (IDT).

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

2016-10-19 Thread Amit Sela (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588301#comment-15588301
 ] 

Amit Sela commented on BEAM-769:


[~kenn] do you think PR#1055 will be ready for merge soon ?

I would generally fix such issue immediately, but it's flakey, only happens if 
something failed (unexpectedly) and will cause a "domino effect" on following 
tests, so I might wait for PR#1055 if we can merge it soon.


> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

2016-10-19 Thread Amit Sela (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590064#comment-15590064
 ] 

Amit Sela commented on BEAM-769:


Merging BEAM-259 which should resolve this, I'll wait a day or two to see it 
does and then close this ticket as duplicate.

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

2016-10-20 Thread Daniel Halperin (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592641#comment-15592641
 ] 

Daniel Halperin commented on BEAM-769:
--

Looks like it has not been resolved yet. There are 3 more flakes today that 
include the fix referenced in BEAM-259. 

https://builds.apache.org/job/beam_PostCommit_MavenVerify/1586/
https://builds.apache.org/job/beam_PostCommit_MavenVerify/1587/
https://builds.apache.org/job/beam_PostCommit_MavenVerify/1588/

Can you take another look?

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

2016-10-20 Thread Amit Sela (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592682#comment-15592682
 ] 

Amit Sela commented on BEAM-769:


Yeah I've noticed, if you look closely you'll see that it's not NPE anymore.
The problem seems to be Jenkins being horribly slow; Spark streaming tests have 
a timeout, and if nothing was processed until a hard timeout was reached, it'll 
rightfully fail.

I'll change the ticket accordingly.

I can set a graceful stop - meaning that the test will finish any currently 
processing batches - but that'll make those tests run X2 or even X3 longer most 
of the time.

Thoughts ?


> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)