How about writing to a buffer ? Then you would flush the buffer to Kafka if and
only if the output operation reports successful completion. In the event of a
worker failure, that would not happen.
—
FG
On Sun, Nov 30, 2014 at 2:28 PM, Josh J wrote:
> Is there a way to do this that preserves
Is there a way to do this that preserves exactly once semantics for the
write to Kafka?
On Tue, Sep 2, 2014 at 12:30 PM, Tim Smith wrote:
> I'd be interested in finding the answer too. Right now, I do:
>
> val kafkaOutMsgs = kafkInMessages.map(x=>myFunc(x._2,someParam))
> kafkaOutMsgs.foreachRDD
I'd be interested in finding the answer too. Right now, I do:
val kafkaOutMsgs = kafkInMessages.map(x=>myFunc(x._2,someParam))
kafkaOutMsgs.foreachRDD((rdd,time) => { rdd.foreach(rec => {
writer.output(rec) }) } ) //where writer.ouput is a method that takes a
string and writer is an instance of a
