[Spark][Core] Resource Allocation

I have some problems that I am looking for if there is no solution for them (due to the current implementation) or if there is a way and I was not aware of it. 1) Currently, we can enable and configure dynamic resource allocation based on below documentation. https://spark.apache.org/docs/late

[Spark] spark client for Hadoop 2.x

e the new Spark clients capable of connecting to the Hadoop 2.x cluster? (According to a simple test, Spark client 3.2.1 had no problem with the Hadoop 2.7 cluster but we wanted to know if there was any guarantee from Spark?) Thank you very much in advance Amin Borjian

[Spark Core] Does Spark support parquet predicate pushdown for big lists?

Hello all, We use Apache Spark 3.2.0 and our data stored on Apache Hadoop with parquet format. One of the advantages of the parquet format is the presence of the predicate pushdown filter feature, which allows only the necessary data to be read. This feature is well provided by Spark. For exam

RE: [Spark] Does Spark support backward and forward compatibility?

) From: Sean Owen<mailto:sro...@gmail.com> Sent: Wednesday, November 24, 2021 10:48 PM To: Amin Borjian<mailto:borjianami...@outlook.com> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: [Spark] Does Spark support backward and forward compatibility? I think

RE: [Spark] Does Spark support backward and forward compatibility?

sro...@gmail.com> Sent: Wednesday, November 24, 2021 5:38 PM To: Amin Borjian<mailto:borjianami...@outlook.com> Cc: user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: [Spark] Does Spark support backward and forward compatibility? Can you mix different Spark versions on drive

[Spark] Does Spark support backward and forward compatibility?

I have a simple question about using Spark, which although most tools usually explain this question explicitly (in important text, such as a specific format or a separate page), I did not find it anywhere. Maybe my search was not enough, but I thought it was good that I ask this question in the

[Spark SQL]: Does Spark SQL can have better performance?

be more restrictive, and it was strange to us that the simple query stated in the email needed such a run time (because with such execution time, heavier queries take longer). Does it make sense that the duration of the query is so long? Is there something we need to pay attention to or can we improve by changing it? Thanks, Amin Borjian