Re: [VOTE] Release Spark 3.2.0 (RC3)

Yes that's correct, the failure is observed with both Hadoop-2.7 as well as Hadoop-2.10 (internal use) On Tue, Sep 21, 2021, 7:15 PM Mridul Muralidharan wrote: > The failure I observed looks the same as what Venkat mentioned, lz4 tests > in FileSuite in core were failing with hadoop-2.7

Re: [VOTE] Release Spark 3.2.0 (RC3)

The failure I observed looks the same as what Venkat mentioned, lz4 tests in FileSuite in core were failing with hadoop-2.7 profile. Regards, Mridul On Tue, Sep 21, 2021 at 7:44 PM Chao Sun wrote: > Hi Venkata, I'm not aware of the FileSuite test failures. In fact I just > tried it locally on

Re: [VOTE] Release Spark 3.2.0 (RC3)

Hi Venkata, I'm not aware of the FileSuite test failures. In fact I just tried it locally on the master branch and the tests are all passing. Could you provide more details? The reason we want to disable the LZ4 test is because it requires the native LZ4 library when running with Hadoop 2.x,

Re: [VOTE] Release Spark 3.2.0 (RC3)

Hi Chao, But there are tests in core as well failing. For eg: org.apache.spark.FileSuite But these tests are passing in 3.1, why do you think we should disable these tests for hadoop version < 3.x? Regards Venkata krishnan On Tue, Sep 21, 2021 at 3:33 PM Chao Sun wrote: > I just created

Re: [VOTE] Release Spark 3.2.0 (RC3)

I just created SPARK-36820 for the above LZ4 test issue. Will post a PR there soon. On Tue, Sep 21, 2021 at 2:05 PM Chao Sun wrote: > Mridul, is the LZ4 failure about Parquet? I think Parquet currently uses > Hadoop compression codec while Hadoop 2.7 still depends on native lib for > the LZ4.

Re: [VOTE] Release Spark 3.2.0 (RC3)

Mridul, is the LZ4 failure about Parquet? I think Parquet currently uses Hadoop compression codec while Hadoop 2.7 still depends on native lib for the LZ4. Maybe we should run the test only for Hadoop 3.2 profile. On Tue, Sep 21, 2021 at 10:08 AM Mridul Muralidharan wrote: > > Signatures,

Re: [VOTE] Release Spark 3.2.0 (RC3)

Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Pmesos -Pkubernetes, this worked fine. I found that including "-Phadoop-2.7" failed on lz4 tests ("native lz4 library not available"). Regards, Mridul On Tue, Sep 21, 2021 at 10:18 AM Gengliang Wang wrote:

Re: [VOTE] Release Spark 3.2.0 (RC3)

To Stephen: Thanks for pointing that out. I agree with that. To Sean: I made a PR to remove the test dependency so that we can start RC4 ASAP. Gengliang On Tue, Sep 21, 2021 at 8:14 PM Sean Owen wrote: > Hm yeah I tend to agree. See

Re: [VOTE] Release Spark 3.2.0 (RC3)

Hm yeah I tend to agree. See https://github.com/apache/spark/pull/33912 This _is_ a test-only dependency which makes it less of an issue. I'm guessing it's not in Maven as it's a small one-off utility; we _could_ just inline the ~100 lines of code in test code instead? On Tue, Sep 21, 2021 at