Code fails when AQE enabled in Spark 3.1

Hi guys, hundreds of spark jobs run on my company every day. We are running Spark 3.1.2 and we want enable Adaptive Query Execution (AQE) for all of them. We can't upgrade to 3.2 right now so we want enable it explicitly using appropriate conf when spark submit. Some of them fails when enable AQE

RE: Does Spark 3.1.2/3.2 support log4j 2.17.1+, and how? your target release day for Spark3.3?

Hi, New releases are announced via mailing lists user@spark.apache.org & d...@spark.apache.org. Best, Meikel From: Theodore J Griesenbrock Sent: Mittwoch, 19. Januar 2022 18:50 To: sro...@gmail.com Cc: Juan Liu ; user@spark.apache.or

RE: Regarding spark-3.2.0 decommission features.

Gentle reminder!!! Br, -Mohan Patidar From: Patidar, Mohanlal (Nokia - IN/Bangalore) Sent: Tuesday, January 18, 2022 2:02 PM To: user@spark.apache.org Cc: Rao, Abhishek (Nokia - IN/Bangalore) ; Gowda Tp, Thimme (Nokia - IN/Bangalore) ; Sharma, Prakash (Nokia - IN/Bangalore) ; Tarun, N (Nokia

Re: Profiling spark application

Give a look at this: https://github.com/LucaCanali/sparkMeasure On 2022/1/20 1:18, Prasad Bhalerao wrote: Is there any way we can profile spark applications which will show no. of invocations of spark api and their execution time etc etc just the way jprofiler shows all the details? -

Re: Profiling spark application

You can take a look at jvm profiler that was open sourced by uber: https://github.com/uber-common/jvm-profiler On Thu, Jan 20, 2022 at 11:20 AM Prasad Bhalerao < prasadbhalerao1...@gmail.com> wrote: > Hi, > > It will require code changes and I am looking at some third party code , I > am lookin

Re: Profiling spark application

Hi, It will require code changes and I am looking at some third party code , I am looking for something which I can just hook to jvm and get the stats.. Thanks, Prasad On Thu, Jan 20, 2022 at 11:00 AM Sonal Goyal wrote: > Hi Prasad, > > Have you checked the SparkListener - > https://mallikarju

Re: Profiling spark application

Hi Prasad, Have you checked the SparkListener - https://mallikarjuna_g.gitbooks.io/spark/content/spark-SparkListener.html ? Cheers, Sonal https://github.com/zinggAI/zingg On Thu, Jan 20, 2022 at 10:49 AM Prasad Bhalerao < prasadbhalerao1...@gmail.com> wrote: > Hello, > > Is there any way we c

Profiling spark application

Hello, Is there any way we can profile spark applications which will show no. of invocations of spark api and their execution time etc etc just the way jprofiler shows all the details? Thanks, Prasad

RE: Does Spark 3.1.2/3.2 support log4j 2.17.1+, and how? your target release day for Spark3.3?

Again, sorry to bother you.   What is the best option available to ensure we get notified when a new version is released for Apache Spark?  I do not see any RSS feeds, nor do I see any e-mail subscription option for this page:  https://spark.apache.org/news/index.html   Please let me know what we c

Re: Does Spark 3.1.2/3.2 support log4j 2.17.1+, and how? your target release day for Spark3.3?

This very user@ list -- announcements will go to all the lists. On Wed, Jan 19, 2022 at 11:50 AM Theodore J Griesenbrock wrote: > Again, sorry to bother you. > > What is the best option available to ensure we get notified when a new > version is released for Apache Spark? I do not see any RSS f

Re: Issue: Spring-Boot vs. Apache Spark Dependencies

The usual answer is to shade dependencies like this in your application, so that both can coexist. On Wed, Jan 19, 2022 at 8:42 AM Heyde, Andreas wrote: > I had only one wish, that someone can have a look at this issue/topic. > > The problem is, that janino 3.0.16 and 3.1.4 has a complete other

AW: Issue: Spring-Boot vs. Apache Spark Dependencies

I had only one wish, that someone can have a look at this issue/topic. The problem is, that janino 3.0.16 and 3.1.4 has a complete other package/file structure, and so they are incompatible. The problem is, that was not done I a major version change, it’s a minor change. So I don’t know, which ve

Re: Issue: Spring-Boot vs. Apache Spark Dependencies

I did not see your message in the moderation queue, not sure why. Indeed you should send to user@, not to individuals. Your message is here now. Do you mean Spark 3.2.0? yes it is on Janino 3.0.16. You can use things like "mvn dependency:tree" on the project build to see dependencies, by the way.

WG: Issue: Spring-Boot vs. Apache Spark Dependencies

see below, got mailing error. Von: Heyde, Andreas Gesendet: Mittwoch, 19. Januar 2022 13:14 An: user@spark.apache.org; user-requ...@spark.apache.org Betreff: Issue: Spring-Boot vs. Apache Spark Dependencies After sending mail to Matei and yesterday subscribing user@spark.apache.org

Issue: Spring-Boot vs. Apache Spark Dependencies

After sending mail to Matei and yesterday subscribing user@spark.apache.org I cannot see my problem in iss...@spark.apache.org, past month - Apache Mail Archives Is there something wrong

Re: Self contained Spark application with local master without spark-submit

Hi Colin, Yes, you can. You only need to define master="local" and build the jar with Main class defined in the manifest. [image: facebook] [image: twitter] [image: linkedin] [image: instagram]

Self contained Spark application with local master without spark-submit

Hello, I noticed I can run spark applications with a local master via sbt run and also via the IDE. I'd like to run a single threaded worker application as a self contained jar. What does sbt run employ that allows it to run a local master? Can I build an uber jar and run without spark-submit?