Re: [VOTE] Release Spark 3.2.0 (RC1)

-1. I found a bug (https://issues.apache.org/jira/browse/SPARK-36558) in the push-based shuffle, which could lead to job hang. Bests, Yi On Sat, Aug 21, 2021 at 1:05 AM Gengliang Wang wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.2.0. > > The vote is

Re: Add option to Spark UI to proxy to the executors?

Oh cool. I’ll have to dig down into why that’s not working with my K8s deployment then. On Sat, Aug 21, 2021 at 11:54 PM Gengliang Wang wrote: > Hi Holden, > > FYI there are already some related features in Spark: > >- Spark Master UI to reverse proxy Application and Workers UI >

Re: [VOTE] Release Spark 3.2.0 (RC1)

Thanks! I found the issue was our explicit dependency on hadoop-client. After dropping that for the one provided by spark-core we no longer run into the Jackson classpath problem. > On Aug 22, 2021, at 1:29 PM, Sean Owen wrote: > > Jackson was bumped from 2.10.x to 2.12.x, which could well

Re: [VOTE] Release Spark 3.2.0 (RC1)

Jackson was bumped from 2.10.x to 2.12.x, which could well explain it if you're exposed to the Spark classpath and have your own different Jackson dep. On Sun, Aug 22, 2021 at 1:21 PM Michael Heuer wrote: > We're seeing runtime classpath issues with Avro 1.10.2, Parquet 1.12.0, > and Spark

Re: [VOTE] Release Spark 3.2.0 (RC1)

We're seeing runtime classpath issues with Avro 1.10.2, Parquet 1.12.0, and Spark 3.2.0 RC1. Our dependency tree is deep though, and will require further investigation. https://github.com/bigdatagenomics/adam/pull/2289 $ mvn test ... *** RUN

Re: [VOTE] Release Spark 3.2.0 (RC1)

So far, I've tested Java 8 + Scala 2.12, Scala 2.13 and the results look good per usual. Good to see Scala 2.13 artifacts!! Unless I've forgotten something we're OK for Scala 2.13 now, and Java 11 (and, IIRC, Java 14 works fine minus some very minor corners of the project's deps) I think we're

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi Gengliang, Yay! Thank you! Java 11 with the following MAVEN_OPTS worked fine: $ echo $MAVEN_OPTS -Xss64m -Xmx4g -XX:ReservedCodeCacheSize=1g $ ./build/mvn \ -Pyarn,kubernetes,hadoop-cloud,hive,hive-thriftserver \ -DskipTests \ clean install ... [INFO]

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi Gengliang, With Java 8 the build worked fine. No other changes. I'm going to give Java 11 a try with the options you mentioned. $ java -version openjdk version "1.8.0_292" OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10,

Re: Add option to Spark UI to proxy to the executors?

Hi Holden, FYI there are already some related features in Spark: - Spark Master UI to reverse proxy Application and Workers UI - Support Spark UI behind front-end reverse proxy using a path prefix Revert proxy URL

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi Mridul, yes, Spark 3.2.0 should include the fix. The PR is merged after the RC1 cut and there is no JIRA for the issue so that it is missed. On Sun, Aug 22, 2021 at 2:27 PM Mridul Muralidharan wrote: > Hi, > > Signatures, digests, etc check out fine. > Checked out tag and build/tested

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi Jacek, The current GitHub action CI for Spark contains Java 11 build. The build is successful with the options "-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g": https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L506 The default Java stack size is small and we have

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes I am seeing test failures which are addressed by #33790 - this is in branch-3.2, but after the RC tag. After updating to the