Re: Implementing circuit breaker pattern in Spark

Are you going to terminate the spark process if the queue is empty? view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or

Re: Implementing circuit breaker pattern in Spark

Hi, >From a technical perspective I think that we all agree, there are no arguments. >From a design/ architecture point of view, given that big data was supposed to solve design challenges on volume, velocity, veracity, and variety, and companies usually investing in data solutions build them to

Re: Implementing circuit breaker pattern in Spark

There's nothing wrong with calling microservices this way. Something needs to call the service with all the data arriving, and Spark is fine for executing arbitrary logic including this kind of thing. Kafka does not change that? On Wed, Feb 16, 2022 at 9:24 AM Gourav Sengupta wrote: > Hi, >

Re: Implementing circuit breaker pattern in Spark

Hi, once again, just trying to understand the problem first. Why are we using SPARK to place calls to micro services? There are several reasons why this should never happen, including costs/ security/ scalability concerns, etc. Is there a way that you can create a producer and put the data into

Re: Implementing circuit breaker pattern in Spark

No I want the job to stop and end once it discovers on repeated retries that the microservice is not responding. But I think I got where you were going right after sending my previous mail. Basically repeatedly failing of your tasks on retries ultimately fails your job anyway. So thats an in-built

Re: Implementing circuit breaker pattern in Spark

You stop the Spark job by tasks failing repeatedly, that's already how it works. You can't kill the driver from the executor other ways, but should not need to. I'm not clear, you're saying you want to stop the job, but also continue processing? On Wed, Feb 16, 2022 at 7:58 AM S wrote: >

Re: Implementing circuit breaker pattern in Spark

Retries have been already implemented. The question is how to stop the spark job by having an executor JVM send a signal to the driver JVM. e.g. I have a microbatch of 30 messages; 10 in each of the 3 partitions. Let's say while a partition of 10 messages was being processed, first 3 went through

Re: Implementing circuit breaker pattern in Spark

You could use the same pattern in your flatMap function. If you want Spark to keep retrying though, you don't need any special logic, that is what it would do already. You could increase the number of task retries though; see the spark.excludeOnFailure.task.* configurations. You can just

Implementing circuit breaker pattern in Spark

Hi, We have a spark job that calls a microservice in the lambda function of the flatmap transformation -> passes to this microservice, the inbound element in the lambda function and returns the transformed value or "None" from the microservice as an output of this flatMap transform. Of course