[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 At least, `spark-master-test-maven-hadoop-2.6` goes green. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Oh, indeed. Thank you for informing that, too! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16795 FYI, I'm fixing OOM in #16825 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Oh, sorry! I misread your comment, `doubt`. For the ReplSuite, #16859 looks related. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16795 Are you asking me? I meant Spark 2.1. Those test errors are not new, that's what I'm saying. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 You mean `avro` or prior `parquet`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16795 > So far, ReplSuite failures are observed. One is OOM... I ran into those issues while testing the 2.1 RCs, so I doubt it's caused by your change. --- If your project is set up for it, you

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 So far, **ReplSuite** failures are observed. One is OOM and the other is the other is ClosureCleaner. Both happens after the testcase, `define case class and create Dataset together with

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Thank you for merging, @srowen . Thank you for review, @vanzin , @liancheng , @jaceklaskowski , too! I'm watching the build system on mockito hotfix. I believe it will become stabler

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-08 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16795 Thanks for the fix, merged to master. we'll see if the build works now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72545/ Test PASSed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72545/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72542/ Test FAILed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72542/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72545/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 The failure seems to be irrelevant to this PR. We can see the result of the next running Jenkins. ``` [INFO] Spark Project Catalyst . FAILURE [11:29 min]

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72537/ Test FAILed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72537 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72537/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72542/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Sure! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72537 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72537/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16795 Yes, maybe a comment too about why this version is specified differently from the parent? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Thank you, @srowen and @vanzin . Then, the only to-do is using 1.8.1 instead of 1.8.0 here? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16795 Hmm, I think this is fine as far as getting the test fixed. If Avro is a well-behaved library, then upgrading it should be fine too (since "well behaved" would mean it maintains backwards

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16795 OK, I get it, so we can't use Avro 1.7.7 with `parquet-avro`. And `parquet-avro` really only is used in test scope. So this change does make sense as-is (though, maybe use version 1.8.1?)

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 The situation here is `apche.parquet.avro.AvroParquetWriter` of `parquet-avro` uses 1.8.0 specific API. Since we use `AvroParquetWriter` in our test suite, we need to use different `avro`

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-07 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Hi, @srowen . I made another PR #16839 for the option 1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 @srowen and @liancheng Could you review this PR again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72479/ Test PASSed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72479 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72479/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72473/ Test FAILed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72473/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72479/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72463/ Test FAILed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72463/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72473/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 The current failure about `ExtendedYarnTest` came from `mesos` module . It seems to be irrelevant to this PR. Let me check that. ``` [info] Running Spark tests using Maven with these

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72467/ Test FAILed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72467/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72464/ Test FAILed. ---

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72464/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72467/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72464/testReport)** for PR 16795 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16795: [SPARK-19409][BUILD][TEST-MAVEN] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72463/testReport)** for PR 16795 at commit