Re: Using Spark as a fail-over platform for Java app

That's what resource managers provide to you. So you can code and deal with resource managers, but I assume you're finding ways to not deal with resource managers directly and let Spark do it instead. I admit I have no experience (I did the similar with Apache Storm on standalone setup 5+ years

Re: Using Spark as a fail-over platform for Java app

Can I cut a steak with a hammer? Sure you can, but the steak would taste awful Do you have organizational/bureaucratic issues with using a Load Balancer? Because that’s what you really need. Run your application on multiple nodes with a load balancer in front. When a node crashes, the load

Using Spark as a fail-over platform for Java app

I have an existing plain-Java (non-Spark) application that needs to run in a fault-tolerant way, i.e. if the node crashes then the application is restarted on another node, and if the application crashes because of internal fault, the application is restarted too. Normally I would run it in a

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Ok, thanks for the clarification. I will try to migrate my project to structured streaming . Regards, Renu On Fri, Mar 12, 2021 at 7:38 PM Gabor Somogyi wrote: > Mainly bugfixes and no breaking AFAIK. > > As a side note there were intentions to close DStreams and discontinue > as-is. > It's

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Mainly bugfixes and no breaking AFAIK. As a side note there were intentions to close DStreams and discontinue as-is. It's not yet happened but it's on the road so I strongly recommend to migrate to Structured Streaming... We simply can't support 2 streaming engines for huge amount of time. G

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Hi Gabor, It seems like it is better to upgrade my spark version . Are there major changes in terms of streaming from spark 2.2 to spark 2.4? PS: I am using KafkaUtils api to create steam Thanks & Regards, Renu yadav On Fri, Mar 12, 2021 at 7:25 PM Renu Yadav wrote: > Thanks Gabor, > This

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Thanks Gabor, This is very useful. Regards, Renu Yadav On Fri, Mar 12, 2021 at 5:36 PM Gabor Somogyi wrote: > Kafka client upgrade is not a trivial change which may or may not work > since new versions can contain incompatible API and/or behavior changes. > I've collected how Spark evolved in

Fwd: compile spark 3.1.1 error

Hi! Zinc cache be cleaned by the shutdown i showed you earlier: ./build/zinc-0.3.15/bin/zinc -shutdown But as I just seen SPARK-34539 : it is absolutely not needed anymore as the standalone zinc is not used by the compiler plugin from v3.0.0.

Re: Issue while consuming message in kafka using structured streaming

Please see that driver side for example resolved in 3.1.0... G On Fri, Mar 12, 2021 at 1:03 PM Sachit Murarka wrote: > Hi Gabor, > > Thanks a lot for the response. I am using Spark 3.0.1 and this is spark > structured streaming. > > Kind Regards, > Sachit Murarka > > > On Fri, Mar 12, 2021 at

Re: How to upgrade kafka client in spark_streaming_kafka 2.2

Kafka client upgrade is not a trivial change which may or may not work since new versions can contain incompatible API and/or behavior changes. I've collected how Spark evolved in terms of Kafka client and there I've gathered the breaking changes to make our life easier. Have a look and based on

Re: Issue while consuming message in kafka using structured streaming

Hi Gabor, Thanks a lot for the response. I am using Spark 3.0.1 and this is spark structured streaming. Kind Regards, Sachit Murarka On Fri, Mar 12, 2021 at 5:30 PM Gabor Somogyi wrote: > Since you've not provided any version I guess you're using 2.x and you're > hitting this issue:

Re: Issue while consuming message in kafka using structured streaming

Since you've not provided any version I guess you're using 2.x and you're hitting this issue: https://issues.apache.org/jira/browse/SPARK-28367 The executor side must be resolved out of the box in the latest Spark version however on driver side one must set "

Issue while consuming message in kafka using structured streaming

Hi All, I am getting following error in spark structured streaming while connecting to Kakfa Main issue from logs:: Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout of 6ms expired before the position for partition my-topic-1 could be determined Current Committed Offsets:

How to upgrade kafka client in spark_streaming_kafka 2.2

Hi Team, I am using spark -2.2 and spark_streamin_kafka 2.2 , which is pointing to kafka-client 0.10 . How can I upgrade a kafka client to kafka 2.2.0 ? Thanks & Regards, Renu Yadav