Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Martin, I see the major benefit of the spark stop() method in giving the ability to shut down the main topic gracefully. I have explained this in this SPIP SPIP: Shutting down spark structured streaming when the streaming process completed current process

Re: Topics for Spark online classes & webinars

Hi Denny, That Apache Spark Linkedin page https://www.linkedin.com/company/apachespark/ looks fine. It also allows a wider audience to benefit from it. +1 for me view my Linkedin profile

Re: Topics for Spark online classes & webinars

In the past, we've been using the Apache Spark LinkedIn page and group to broadcast these type of events - if you're cool with this? Or we could go through the process of submitting and updating the current https://spark.apache.org or request to

unsubscribe

unsubscribe

Re: Topics for Spark online classes & webinars

+! For these classes, Please share confluence page link. -Manisha From: asma zgolli Date: Monday 13 March 2023 at 16:30 To: Mich Talebzadeh Cc: "user @spark" , Spark dev list Subject: RE: [EXTERNAL]Topics for Spark online classes & webinars CAUTION: This email originated from outside of the

Re: Adding pause() method to pyspark.sql.streaming.StreamingQuery

Hi Mich. I'm trying to understand, can you please provide some use-cases where it would be beneficial with a pause and how a pause would differ functionally from a stop? Best regards, Martin From: Mich Talebzadeh Sent: Thursday, March 9, 2023 17:12 To: Spark

Re: spark executor pod has same memory value for request and limit

There is a very good reason for this. It is recommended using k8s that you set memory request and limit to the same value, set a cpu request, but not a cpu limit. More info here https://home.robusta.dev/blog/kubernetes-memory-limit BR, Martin From: Mich