Re: Spark on K8s - repeating annoying exception

2022-05-13 Thread Martin Grigorov
Hi, On Mon, May 9, 2022 at 5:57 PM Shay Elbaz wrote: > Hi all, > > > > I apologize for reposting this from Stack Overflow, but it got very little > attention and now comment. > > > > I'm using Spark 3.2.1 image that was built from the official distribution > via `docker-image-tool.sh', on

Re: Spark on K8s , some applications ended ungracefully

2022-04-01 Thread Martin Grigorov
Hi, On Thu, Mar 31, 2022 at 4:18 PM Pralabh Kumar wrote: > Hi Spark Team > > Some of my spark applications on K8s ended with the below error . These > applications though completed successfully (as per the event log > SparkListenerApplicationEnd event at the end) > stil have even files with

Re: spark distribution build fails

2022-03-17 Thread Martin Grigorov
Hi, For the mail archives: this error happens when the user has MAVEN_OPTS env var pre-exported. In this case ./build/mvn|sbt does not export its own MAVEN_OPTS with the -XssXYZ value, and the default one is too low and leads to the StackOverflowError On Mon, Mar 14, 2022 at 11:13 PM

Re: how can I remove the warning message

2022-02-04 Thread Martin Grigorov
Hi, This is a JVM warning, as Sean explained. You cannot control it via loggers. You can disable it by passing --illegal-access=permit to java. Read more about it at https://softwaregarden.dev/en/posts/new-java/illegal-access-in-java-16/ On Sun, Jan 30, 2022 at 4:32 PM Sean Owen wrote: > This

Re: Spark 3.1 Json4s-native jar compatibility

2022-02-04 Thread Martin Grigorov
Hi, Amit said that he uses Spark 3.1, so the link should be https://github.com/apache/spark/blob/branch-3.1/pom.xml#L879 (3.7.0-M5) @Amit: check your classpath. Maybe there are more jars of this dependency. On Thu, Feb 3, 2022 at 10:53 PM Sean Owen wrote: > You can look it up: >

Re: [EXTERNAL] Fwd: Log4j upgrade in spark binary from 1.2.17 to 2.17.1

2022-01-31 Thread Martin Grigorov
Hi, On Mon, Jan 31, 2022 at 7:57 PM KS, Rajabhupati wrote: > Thanks a lot Sean. One final question before I close the conversion how do > we know what are the features that will be added as part of spark 3.3 > version? > There will be release notes for 3.3 at linked at