Re: spark 3.2 release date

Hi, There is not exact release date now. As per https://spark.apache.org/release-process.html , we need a Release Candidate which passes the release vote. Spark 3.2 RC1 failed recently. I will cut RC2 after

spark 3.2 release date

What is the expected ballpark release date of spark 3.2 ? Thanks and Regards, Ajay.

Re: Can’t write to PVC in K8S

ok, so when I use spark on k8s I can only save files to s3 buckets or to a database? Note my setup, its spark with jupyterlab on top on k8s. What are those for if I cant write files from spark in k8s to disk? "spark.kubernetes.driver.volumes.persistentVolumeClaim.nfs100.mount.readOnly",

Re: Performance Degradation in Spark 3.0.2 compared to Spark 3.0.1

The problem with these tickets is that it tends to generalise the performance as opposed to a statement of specifics. According to the latter ticket it states and I quote "Spark 3.1.1 is slower than 3.0.2 by 4-5 times". This is not what we have observed migrating from 3.0.1 to 3.1.1. Unless it

Re: Connection reset by peer : failed to remove cache rdd

Hi, No idea what might be going on here, but I'd not worry much about it and simply monitor disk usage as some broadcast blocks might have left over. Do you know when in your application lifecycle it happens? Spark SQL or Structured Streaming? Do you use broadcast variables or are the errors

Connection reset by peer : failed to remove cache rdd

We are facing issue in production where we are getting frequent Still have 1 request outstanding when connection with the hostname was closed connection reset by peer : errors as well as warnings : failed to remove cache rdd or failed to remove broadcast variable. Please help us how to

Can’t write to PVC in K8S

Hi, I have built and running spark on k8s. A link to my repo https://github.com/bjornjorgensen/jlpyk8s Everything seems to be running fine, but I can’t save to PVC. If I convert the dataframe to pandas, then I can save it. from pyspark.sql import SparkSession spark = SparkSession.builder \

Unsubscribe

Unsubscribe

Unsubscribe

Re: Performance Degradation in Spark 3.0.2 compared to Spark 3.0.1

Hi , we are not moving to 3.1.1 because some open ticket are there I have mentioned below. https://issues.apache.org/jira/browse/SPARK-30536 https://issues.apache.org/jira/browse/SPARK-35066 please refer attached mail for spark 35066. Thanks. From: Mich

Re: Performance Degradation in Spark 3.0.2 compared to Spark 3.0.1

Hi, Any particular reason why you are not using 3.1.1 on Kubernetes? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property

Unsubscribe

Unsubscribe

- To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe