Thanks Ameya.
From: ameya [via Apache Spark User List]
[mailto:ml-node+s1001560n24650...@n3.nabble.com]
Sent: Friday, September 11, 2015 4:12 AM
To: Samya MAITI
Subject: Re: Maintaining Kafka Direct API Offsets
So I added something like this:
Runtime.getRuntime().addShutdownHook(new Thread
Hi Ameya,
Plz suggest, when you say graceful shut-down, what exactly did you handle?
Thanks.
Thanks,
Sam
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Maintaining-Kafka-Direct-API-Offsets-tp24246p24636.html
Sent from the Apache Spark User List mailing l
Thanks. Looking at the KafkaCluster.scala code, (
https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaCluster.scala#L253),
it seems a little hacky for me to alter and recompile spark to expose those
methods, so I'll use the receiver API fo
I don't entirely agree with that assessment. Not paying for extra cores to
run receivers was about as important as delivery semantics, as far as
motivations for the api.
As I said in the jira tickets on the topic, if you want to use the direct
api and save offsets to ZK, you can. The right way
In summary, it appears that the use of the DirectAPI was intended
specifically to enable exactly-once semantics. This can be achieved for
idempotent transformations and with transactional processing using the
database to guarantee an "onto" mapping of results based on inputs. For the
latter, you ne
For those who find this post and may be interested, the most thorough
documentation on the subject may be found here:
https://github.com/koeninger/kafka-exactly-once
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Maintaining-Kafka-Direct-API-Offsets-tp24246
Use your email client to send a message to the mailing list from the email
address you used to subscribe?
The message you just sent reached the list
On Fri, Aug 14, 2015 at 9:36 AM, dutrow wrote:
> How do I get beyond the "This post has NOT been accepted by the mailing
> list
> yet" message? Th
How do I get beyond the "This post has NOT been accepted by the mailing list
yet" message? This message was posted through the nabble interface; one
would think that would be enough to get the message accepted.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com
