Re: Add option to Spark UI to proxy to the executors?

Yes I can see your point. Will that work in kubernetes deployment? Mich view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property w

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, I've been building the tag and I'm facing the following StackOverflowError: Exception in thread "main" java.lang.StackOverflowError at scala.tools.nsc.transform.ExtensionMethods$Extender.transform(ExtensionMethods.scala:275) at scala.tools.nsc.transform.ExtensionMethods$Extender.transform(Ext

Re: Spark docker image convention proposal for docker repository

Hey Mich Adding to what Mich is suggesting, how about having the base OS version in the image tag as well, like --- 3.1.2-scala_2.12-java11-slim 3.1.2_sparkpy-scala_2.12-java11-buster 3.1.2_sparkR-scala_2.12-java11-slim Regards. Ankit Prakash Gupta info.ank...@gmail.com LinkedIn : https://www.

Re: Spark docker image convention proposal for docker repository

Hi Ankit, Sure I suppose that elaboration on OS base can be added BASE_OS="buster" SPARK_VERSION="3.1.1" SCALA_VERSION="scala_2.12" DOCKERFILE="Dockerfile" DOCKERIMAGETAG="11-jre-slim" # Building Docker image from provided Dockerfile base 11 cd $SPARK_HOME /opt/spark/bin/docker-image-tool.sh \

Re: Spark docker image convention proposal for docker repository

Sorry there was a typo BASE_OS="buster" SPARK_VERSION="3.1.1" SCALA_VERSION="scala_2.12" DOCKERFILE="Dockerfile" DOCKERIMAGETAG="11-jre-slim" cd $SPARK_HOME # Building Docker image from provided Dockerfile base 11 cd $SPARK_HOME /opt/spark/bin/docker-image-tool.sh \ -r spark -t ${SP

Re: Spark docker image convention proposal for docker repository

Apologies ignore the first line spark/spark-py (that is redundant). These are correct REPOSITORY TAG IMAGE ID CREATED SIZE spark/spark 3.1.1-scala_2.12-11-jre-slim-buster 71ff5ed3ca03 9 seconds ago 635MB openjdk 8-jre-slim

Re: Spark docker image convention proposal for docker repository

Hi all, These docker images are created on spark 3.1.1. The reason I have chosen this version for now is that most production offerings (for example Google Dataproc) are based on 3.1.1. So with 3.1.1, we have scala-2.12 with Java version 11-jre-slim OR Java version 8-jre-slim all currently on D

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi, Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Pmesos -Pkubernetes I am seeing test failures which are addressed by #33790 - this is in branch-3.2, but after the RC tag. After updating to the h

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi Jacek, The current GitHub action CI for Spark contains Java 11 build. The build is successful with the options "-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g": https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L506 The default Java stack size is small and we have to

Re: [VOTE] Release Spark 3.2.0 (RC1)

Hi Mridul, yes, Spark 3.2.0 should include the fix. The PR is merged after the RC1 cut and there is no JIRA for the issue so that it is missed. On Sun, Aug 22, 2021 at 2:27 PM Mridul Muralidharan wrote: > Hi, > > Signatures, digests, etc check out fine. > Checked out tag and build/tested with

Re: Add option to Spark UI to proxy to the executors?

Hi Holden, FYI there are already some related features in Spark: - Spark Master UI to reverse proxy Application and Workers UI - Support Spark UI behind front-end reverse proxy using a path prefix Revert proxy URL