Re: Welcome Yikun Jiang as a Spark committer

Congratulations, Yikun! On Sat, Oct 8, 2022 at 7:41 AM Hyukjin Kwon wrote: > Hi all, > > The Spark PMC recently added Yikun Jiang as a committer on the project. > Yikun is the major contributor of the infrastructure and GitHub Actions in > Apache Spark as well as Kubernates and PySpark. > He has

Re: [VOTE] SPIP: Support Docker Official Image for Spark

+1 On Thu, Sep 22, 2022, 04:42 Hyukjin Kwon wrote: > Hi all, > > I would like to start a vote for SPIP: "Support Docker Official Image for > Spark" > > The goal of the SPIP is to add Docker Official Image(DOI) > to ensure the Spark > Docker ima

Re: [DISCUSS] SPIP: Support Docker Official Image for Spark

+1 Good idea! Martin On Mon, Sep 19, 2022 at 3:16 AM Yikun Jiang wrote: > Hi, all > > I would like to start the discussion for supporting Docker Official Image > for Spark. > > This SPIP is proposed to add Docker Official Image(DOI) > to ensu

Re: [VOTE] Release Spark 3.3.0 (RC6)

Hi, [X] +1 Release this package as Apache Spark 3.3.0 Tested: - make local distribution from sources (with ./dev/make-distribution.sh --tgz --name with-volcano -Pkubernetes,volcano,hadoop-3) - create a Docker image (with JDK 11) - run Pi example on -- local -- Kubernetes with default scheduler --

Re: Spark32 + Java 11 . Reading parquet java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'

Hi Pralabh, The Dockerfile defines and ARG for the JDK version: https://github.com/apache/spark/blob/861df43e8d022f51727e0a12a7cca5e119e3c4cc/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile#L17 That means you could use --build-arg to overwrite it when building the image.

Re: [VOTE] Release Spark 3.3.0 (RC5)

Hi, [X] +1 Release this package as Apache Spark 3.3.0 Tested: - make local distribution from sources (with ./dev/make-distribution.sh --tgz --name with-volcano -Pkubernetes,volcano,hadoop-3) - create a Docker image (with JDK 11) - run Pi example on -- local -- Kubernetes with default scheduler --

Re: [VOTE] Release Spark 3.3.0 (RC2)

Hi, [X] +1 Release this package as Apache Spark 3.3.0 Tested: - make local distribution from sources (with ./dev/make-distribution.sh --tgz --name with-volcano -Pkubernetes,volcano,hadoop-3) - create a Docker image (with JDK 11) - run Pi example on -- local -- Kubernetes with default scheduler --

Re: CVE-2020-13936

Hi, On Thu, May 5, 2022 at 8:44 PM Sean Owen wrote: > This is a Velocity issue. Spark doesn't use it, although it looks like > Avro does. From reading the CVE, I do not believe it would impact Avro's > usage - velocity templates it may use for codegen aren't exposed that I > know of. Is there a

Re: Is spark fair scheduler is for kubernete?

Hi, On Mon, Apr 11, 2022 at 7:43 AM Jason Jun wrote: > the official doc, https://spark.apache.org/docs/latest/job-scheduling.html, > didn't mention that its working for kubernete cluster? > You could use Volcano scheduler for more advanced setups on Kubernetes. Here is an article explaining ho

Re: Spark on K8s , some applications ended ungracefully

Hi, On Thu, Mar 31, 2022 at 4:18 PM Pralabh Kumar wrote: > Hi Spark Team > > Some of my spark applications on K8s ended with the below error . These > applications though completed successfully (as per the event log > SparkListenerApplicationEnd event at the end) > stil have even files with .inp

Re: Problem building spark-catalyst_2.12 with Maven

t quite understand it. Try the stack size in test runner > configs > > On Thu, Feb 10, 2022, 2:02 PM Martin Grigorov > wrote: > >> Hi Sean, >> >> On Thu, Feb 10, 2022 at 5:37 PM Sean Owen wrote: >> >>> Yes I've seen this; the JVM stack size needs

Re: Problem building spark-catalyst_2.12 with Maven

igger values for all jvmArgs next to this one. None helped! I also have the feeling it is something in my environment that overrides these values but so far I cannot identify anything. > > On Thu, Feb 10, 2022 at 8:54 AM Martin Grigorov > wrote: > >> Hi, >> >>

Problem building spark-catalyst_2.12 with Maven

Hi, I am not able to build Spark due to the following error : ERROR] ## Exception when compiling 543 sources to /home/martin/git/apache/spark/sql/catalyst/target/scala-2.12/classes java.lang.BootstrapMethodError: call site initialization exception java.lang.invoke.CallSite.makeSite(CallSite.java:

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

On Wed, Apr 7, 2021 at 3:41 PM Hyukjin Kwon wrote: > Hi Greg, > > I raised this thread to figure out a way that we can work together to > resolve this issue, gather feedback, and to understand how other projects > work around. > Several projects I observed, as far as I can tell, have made enough