Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

have a mechanism to limit the number of >>>> retries for executor recreation. If the system fails to successfully create >>>> an executor more than a specified number of times (e.g., 5 attempts), the >>>> entire Spark application should fail and stop trying

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

ge retry configurations > (`spark.task.maxFailures`, `spark.stage.maxConsecutiveAttempts`), but these > do not directly address the issue of limiting executor creation retries. > Implementing a custom monitoring solution to track executor failures and > manually stop the application

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

ilure states. >>> >>> *Questions for the Community* >>> >>> 1. Is there an existing configuration or method within Spark or the >>> Spark Operator to limit executor recreation attempts and fail the job after >>> reaching a threshold? >>&g

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

ve explored Spark's task and stage retry configurations >> (`spark.task.maxFailures`, `spark.stage.maxConsecutiveAttempts`), but these >> do not directly address the issue of limiting executor creation retries. >> Implementing a custom monitoring solution to track executor failures and >> manually stop the application is a potential workaround, but it would be >> preferable to have a more integrated solution. >> >> I appreciate any guidance, insights, or feedback you can provide on this >> matter. >> >> Thank you for your time and support. >> >> Best regards, >> Sri P >> >

Re: [Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

gt; I have explored Spark's task and stage retry configurations > (`spark.task.maxFailures`, `spark.stage.maxConsecutiveAttempts`), but these > do not directly address the issue of limiting executor creation retries. > Implementing a custom monitoring solution to track executor failure

[Spark on Kubernetes]: Seeking Guidance on Handling Persistent Executor Failures

potential workaround, but it would be preferable to have a more integrated solution. I appreciate any guidance, insights, or feedback you can provide on this matter. Thank you for your time and support. Best regards, Sri P

Re: Seeking Guidance on Spark on Kubernetes Secrets Configuration

t does, any elucidation on the method or best practices would be pivotal for our project. Alternatively, if you could point me to resources or community experts who have tackled similar challenges, it would be of immense assistance. Thank you for bearing with the intricacies of our query, and I ap

Re: Seeking Guidance on Spark on Kubernetes Secrets Configuration

would be of immense assistance.Thank you for bearing with the intricacies of our query, and I appreciate your continued guidance in this endeavor.Warm regards,Jon Rodríguez Aranguren.El sáb, 30 sept 2023 a las 23:19, Jayabindu Singh (<jayabi...@gmail.com>) escribió:Hi Jon,Using IAM as suggested

Re: Seeking Guidance on Spark on Kubernetes Secrets Configuration

ication. My objective is to understand the best approach or methods to ensure that these secrets can be smoothly accessed by the Spark application.If any of you have previously encountered this scenario or possess relevant insights on the matter, your guidance would be highly beneficial.Thank you for

Re: Seeking Guidance on Spark on Kubernetes Secrets Configuration

configure multiple Kubernetes secrets, notably >> multiple S3 keys, at the SparkConf level for a Spark application. My >> objective is to understand the best approach or methods to ensure that >> these secrets can be smoothly accessed by the Spark application. >> >> If

Re: Seeking Guidance on Spark on Kubernetes Secrets Configuration

ion. My > objective is to understand the best approach or methods to ensure that > these secrets can be smoothly accessed by the Spark application. > > If any of you have previously encountered this scenario or possess > relevant insights on the matter, your guidance would be highly

Re: Seeking Guidance on Spark on Kubernetes Secrets Configuration

eys, at the SparkConf level for a Spark application. My > objective is to understand the best approach or methods to ensure that these > secrets can be smoothly accessed by the Spark application. > > If any of you have previously encountered this scenario or possess relevant > insights o

Seeking Guidance on Spark on Kubernetes Secrets Configuration

or methods to ensure that these secrets can be smoothly accessed by the Spark application. If any of you have previously encountered this scenario or possess relevant insights on the matter, your guidance would be highly beneficial. Thank you for your time and consideration. I'm eager to

Re: Urgent: Seeking Guidance on Kafka Slow Consumer and Data Skew Problem

ed similar > issues. > > > Thanks and regards, > Gowtham S > > > On Tue, 19 Sept 2023 at 17:23, Karthick > wrote: > >> Subject: Seeking Guidance on Kafka Slow Consumer and Data Skew Problem >> >> Dear Spark Community, >> >> I recently reached ou

Re: Urgent: Seeking Guidance on Kafka Slow Consumer and Data Skew Problem

em, please feel free to share them. Looking forward to hearing from others who might have encountered similar issues. Thanks and regards, Gowtham S On Tue, 19 Sept 2023 at 17:23, Karthick wrote: > Subject: Seeking Guidance on Kafka Slow Consumer and Data Skew Problem > > Dear Spark C

Urgent: Seeking Guidance on Kafka Slow Consumer and Data Skew Problem

Subject: Seeking Guidance on Kafka Slow Consumer and Data Skew Problem Dear Spark Community, I recently reached out to the Apache Flink community for assistance with a critical issue we are facing in our IoT platform, which relies on Apache Kafka and real-time data processing. We received some

Fwd: 📅 Wednesday: Join 6 Members at "Ofir Press | Complementing Scale: Novel Guidance Methods for Improving LMs"

destruction. -- Forwarded message - From: Apache Spark+AI London Date: Thu, 24 Aug 2023 at 20:01 Subject: 📅 Wednesday: Join 6 Members at "Ofir Press | Complementing Scale: Novel Guidance Methods for Improving LMs" To: Apache Spark+AI London invites you to keep connecting

Guidance

Hi everyone, I want to ask for guidance for my log analyzer platform idea. I have an elasticsearch system which collects the logs from different platforms, and creates alerts. The system writes the alerts to an index on ES. Also, my alerts are stored in a folder as JSON (multi line format). The

Re: Need some guidance

Thanks, yes. I was using Int for my V and didn't get the second param in the second closure right :) On Mon, Apr 13, 2015 at 1:55 PM, Dean Wampler wrote: > That appears to work, with a few changes to get the types correct: > > input.distinct().combineByKey((s: String) => 1, (agg: Int, s: String)

Re: Need some guidance

That appears to work, with a few changes to get the types correct: input.distinct().combineByKey((s: String) => 1, (agg: Int, s: String) => agg + 1, (agg1: Int, agg2: Int) => agg1 + agg2) dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition

Re: Need some guidance

How about this? input.distinct().combineByKey((v: V) => 1, (agg: Int, x: Int) => agg + 1, (agg1: Int, agg2: Int) => agg1 + agg2).collect() On Mon, Apr 13, 2015 at 10:31 AM, Dean Wampler wrote: > The problem with using collect is that it will fail for large data sets, > as you'll attempt to copy

Re: Need some guidance

The problem with using collect is that it will fail for large data sets, as you'll attempt to copy the entire RDD to the memory of your driver program. The following works (Scala syntax, but similar to Python): scala> val i1 = input.distinct.groupByKey scala> i1.foreach(println) (1,CompactBuffer(b

Need some guidance

**Learning the ropes** I'm trying to grasp the concept of using the pipeline in pySpark... Simplified example: >>> list=[(1,"alpha"),(1,"beta"),(1,"foo"),(1,"alpha"),(2,"alpha"),(2,"alpha"),(2,"bar"),(3,"foo")] Desired outcome: [(1,3),(2,2),(3,1)] Basically for each key, I want the number of un

Re: guidance on simple unit testing with Spark

iously, I have written unit tests using specs2 framework > and > > have got them to work in Scalding. I tried to use the specs2 framework > with > > Spark, but could not find any simple examples I could follow. I am open > to > > specs2 or Funsuite, whichever wo

Re: guidance on simple unit testing with Sprk

good understanding of writing unit tests using the > Spark > > framework. Previously, I have written unit tests using specs2 framework > and > > have got them to work in Scalding. I tried to use the specs2 framework > with > > Spark, but could not find any simple examples

Re: guidance on simple unit testing with Spark

he specs2 framework with > Spark, but could not find any simple examples I could follow. I am open to > specs2 or Funsuite, whichever works best with Spark. I would like some > additional guidance, or some simple sample code using specs2 or Funsuite. My > code is provided below. > >

guidance on simple unit testing with Spark

unit tests using specs2 framework and have got them to work in Scalding. I tried to use the specs2 framework with Spark, but could not find any simple examples I could follow. I am open to specs2 or Funsuite, whichever works best with Spark. I would like some additional guidance, or some simple