RE: Regarding spark-3.2.0 decommission features.

: Regarding spark-3.2.0 decommission features. Gentle reminder!!! Br, -Mohan Patidar From: Patidar, Mohanlal (Nokia - IN/Bangalore) Sent: Tuesday, January 18, 2022 2:02 PM To: user@spark.apache.org<mailto:user@spark.apache.org> Cc: Rao, Abhishek (Nokia - IN/Bangalore) mailto:abhishek@nok

RE: Inclusive terminology usage in Spark

HI Sean, Thanks for the quick response. We’ll look into this. Thanks and Regards, Abhishek From: Sean Owen Sent: Wednesday, June 30, 2021 6:30 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Cc: User Subject: Re: Inclusive terminology usage in Spark This was covered and mostly done last year

Inclusive terminology usage in Spark

Hi, Terms such as Blacklist/Whitelist and master/slave is used at different places in Spark Code. Wanted to know if there are any plans to modify this to more inclusive terminology, for eg: Denylist/Allowlist and Leader/Follower? If so, what is the timeline? I've also created an improvement

RE: Why is Spark 3.0.x faster than Spark 3.1.x

Hi Maziyar, Mich Do we have any ticket to track this? Any idea if this is going to be fixed in 3.1.2? Thanks and Regards, Abhishek From: Mich Talebzadeh Sent: Friday, April 9, 2021 2:11 PM To: Maziyar Panahi Cc: User Subject: Re: Why is Spark 3.0.x faster than Spark 3.1.x Hi, Regarding

s3a staging committer (directory committer) not writing data to s3 bucket (final output directory) in spark3

Hi, I'm running spark3 on Kubernetes and using S3A staging committer (directory committer) to write data to s3 bucket. The same set up works fine with spark 2.4.5 but with spark3 the final data (writing in parquet format) is not visible in s3 bucket and when read operation is performed on that

RE: Spark 3.0 using S3 taking long time for some set of TPC DS Queries

were seeing discrepancy in query execution time on S3 with Spark 3.0.0. Thanks and Regards, Abhishek From: Gourav Sengupta Sent: Wednesday, August 26, 2020 5:49 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Cc: user Subject: Re: Spark 3.0 using S3 taking long time for some set of TPC DS Queries

RE: Spark 3.0 using S3 taking long time for some set of TPC DS Queries

Yeah… Not sure if I’m missing any configurations which is causing this issue. Any suggestions? Thanks and Regards, Abhishek From: Gourav Sengupta Sent: Wednesday, August 26, 2020 2:35 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Cc: user@spark.apache.org Subject: Re: Spark 3.0 using S3 taking

RE: Spark 3.0 using S3 taking long time for some set of TPC DS Queries

Hi Gourav, Yes. We’re using s3a. Thanks and Regards, Abhishek From: Gourav Sengupta Sent: Wednesday, August 26, 2020 1:18 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Cc: user@spark.apache.org Subject: Re: Spark 3.0 using S3 taking long time for some set of TPC DS Queries Hi, are you using

RE: Spark 3.0 using S3 taking long time for some set of TPC DS Queries

whereas in case of HDFS, it is only 4.5 GB. Any idea why this difference is there? Thanks and Regards, Abhishek From: Luca Canali Sent: Monday, August 24, 2020 7:18 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Cc: user@spark.apache.org Subject: RE: Spark 3.0 using S3 taking long time for some set

RE: Spark Thrift Server in Kubernetes deployment

Hi, STS deployment on k8s is not supported out of the box. We had done some minor changes in spark code to get Spark Thrift Server working on k8s. Here is the PR that we had created. https://github.com/apache/spark/pull/22433 Unfortunately, this could not be merged. Thanks and Regards,

RE: [External Sender] Spark Executor pod not getting created on kubernetes cluster

Hi Manish, Is this issue resolved? If not, please check the overlay network of your cluster once. We had faced similar issues when we had problems with overlay networking. In our case, executor had spawned, but the communication with driver and executor had failed (due to issues with overlay

RE: web access to sparkUI on docker or k8s pods

Hi, We have seen this issue when we tried to bringup the UI on custom ingress path (default ingress path “/” works). Do you also have similar configuration? We tired setting spark.ui.proxyBase and spark.ui.reverseProxy but did not help. As a workaround, we’re using ingress port (port on edge

RE: Spark on Kubernetes - log4j.properties not read

Hi Dave, As part of driver pod bringup, a configmap is created using all the spark configuration parameters (with name spark.properties) and mounted to /opt/spark/conf. So all the other files present in /opt/spark/conf will be overwritten. Same is happening with the log4j.properties in this

RE: Spark UI History server on Kubernetes

spark.eventLog.dir Thanks and Regards, Abhishek From: Battini Lakshman Sent: Wednesday, January 23, 2019 1:55 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Subject: Re: Spark UI History server on Kubernetes HI Abhishek, Thank you for your response. Could you please let me know the properties you configured

RE: Spark UI History server on Kubernetes

Hi, We’ve setup spark-history service (based on spark 2.4) on K8S. UI works perfectly fine when running on NodePort. We’re facing some issues when on ingress. Please let us know what kind of inputs do you need? Thanks and Regards, Abhishek From: Battini Lakshman Sent: Tuesday, January 22,