Re: [Spark-Core] Improving Reliability of spark when Executors OOM

Hi all, Sorry for the delay in getting the first draft of (my first) SPIP out. https://docs.google.com/document/d/1hxEPUirf3eYwNfMOmUHpuI5dIt_HJErCdo7_yr9htQc/edit?pli=1 Let me know what you think. Regards kalyan. On Sat, Jan 20, 2024 at 8:19 AM Ashish Singh wrote: > Hey all, > > Thanks for

Re: [EXTERNAL] Re: Spark Kafka Rack Aware Consumer

Overall the proposal to make this an option for Kafka source SGTM. You can address the doc review and can send PR (in parallel or after the review). Note that currently executors cache client connection to Kafka and reuse the connection and buffered records for next micro-batch. Your proposal

Re: [EXTERNAL] Re: Spark Kafka Rack Aware Consumer

Granted. Thanks for bearing with me. I’ve also opened up permissions to allow anyone with the link to edit the document. Thank you! From: Mich Talebzadeh Date: Friday, January 26, 2024 at 09:19 To: "Schwager, Randall" Cc: "dev@spark.apache.org" Subject: Re: [EXTERNAL] Re: Spark Kafka Rack

Re: [EXTERNAL] Re: Spark Kafka Rack Aware Consumer

Ok I made a request to access this document Thanks Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile Ent https://en.everybodywiki.com/Mich_Talebzadeh

Re: [EXTERNAL] Re: Spark Kafka Rack Aware Consumer

Hi Mich, Thanks for responding. In the JIRA issue, the design doc you’re referring to describes the prior work. This is the design doc for the proposed change: https://docs.google.com/document/d/1RoEk_mt8AUh9sTQZ1NfzIuuYKf1zx6BP1K3IlJ2b8iM/edit#heading=h.pbt6pdb2jt5c I’ll re-word the

Re: Spark Kafka Rack Aware Consumer

Your design doc Structured Streaming Kafka Source - Design Doc - Google Docs seems to be around since 2016. Reading the comments it was decided not to progress with it. What has changed