Sync commit to kafka 0.10

2017-08-30 Thread krot.vyacheslav
Hi, I'm looking for a way to make a sync commit of offsets to kafka 0.10? commitAsync works well, but I'd like to proceed to next job only after successful commit, a small additional latency is not an issue for my usecase. I know I can store offsets somewhere else, but builtin kafka offset

Storing a lot of state with updateStateByKey

2015-05-14 Thread krot.vyacheslav
Hi all, I'm a complete newbie to spark and spark streaming, so the question may seem obvious, sorry for that. It is okay to store Seq[Data] in state when using 'updateStateByKey'? I have a function with signature def saveState(values: Seq[Msg], value: Option[Iterable[Msg]]): Option[Iterable[Msg]]