Re: [Spark Structured Streaming]: Dynamic Scaling of Executors

Hi, Thanks for your response. I understand there is no explicit way to configure dynamic scaling for Spark Structured Streaming as the ticket is still open for that. But is there a way to manage dynamic scaling with the existing Batch Dynamic scaling algorithm as this kicks in when Dynamic

Re: Re: maven with Spark 3.4.0 fails compilation

2.13.8 you must change 2.13.6 to 2.13.8 man. 29. mai 2023 kl. 18:02 skrev Mich Talebzadeh : > Thanks everyone. Still not much progress :(. It is becoming a bit > confusing as I am getting

Re: Re: maven with Spark 3.4.0 fails compilation

Thanks everyone. Still not much progress :(. It is becoming a bit confusing as I am getting this error Compiling ReduceByKey [INFO] Scanning for projects... [INFO] [INFO] -< spark:ReduceByKey >-- [INFO] Building ReduceByKey 3.0 [INFO] from pom.xml

Re: Re: maven with Spark 3.4.0 fails compilation

Change org.scala-lang scala-library 2.13.11-M2 to org.scala-lang scala-library ${scala.version} man. 29. mai 2023 kl. 13:20 skrev Lingzhe Sun : > Hi Mich, > > Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8 >

Re: JDK version support information

Per docs, it is Java 8. It's possible Java 11 partly works with 2.x but not supported. But then again 2.x is not supported either. On Mon, May 29, 2023, 6:43 AM Poorna Murali wrote: > We are currently using JDK 11 and spark 2.4.5.1 is working fine with that. > So, we wanted to check the maximum

Re: JDK version support information

We are currently using JDK 11 and spark 2.4.5.1 is working fine with that. So, we wanted to check the maximum JDK version supported for 2.4.5.1. On Mon, 29 May, 2023, 5:03 pm Aironman DirtDiver, wrote: > Spark version 2.4.5.1 is based on Apache Spark 2.4.5. According to the > official Spark

Re: JDK version support information

Spark version 2.4.5.1 is based on Apache Spark 2.4.5. According to the official Spark documentation for version 2.4.5, the maximum supported JDK (Java Development Kit) version is JDK 8 (Java 8). Spark 2.4.5 is not compatible with JDK versions higher than Java 8. Therefore, you should use JDK 8 to

JDK version support information

Hi, We are using spark version 2.4.5.1. We would like to know the maximum JDK version supported for the same. Thanks, Poorna

Re: Re: maven with Spark 3.4.0 fails compilation

Hi Mich, Spark 3.4.0 prebuilt with scala 2.13 is built with version 2.13.8. Since you are using spark-core_2.13 and spark-sql_2.13, you should stick to the major(13) and the minor version(8). Not using any of these may cause unexpected behaviour(though scala claims compatibility among minor

Re: maven with Spark 3.4.0 fails compilation

Thanks for your helpful comments Bjorn. I managed to compile the code with maven but when it run it fails with Application is ReduceByKey Exception in thread "main" java.lang.NoSuchMethodError: scala.package$.Seq()Lscala/collection/immutable/Seq$; at