Re: Samza 0.12.0 + synchronous KafkaProducer ?

l checkpoint after > > > every send() call inside same process() call.. that seems to be > > > problematic, as once checkpointed, there is no safety net against any > > > failures in subsequent sends() from same process call. > > > > > > Thanks for being patient with my questions!

Re: Samza 0.12.0 + synchronous KafkaProducer ?

t seems to be > > > problematic, as once checkpointed, there is no safety net against any > > > failures in subsequent sends() from same process call. > > > > > > Thanks for being patient with my questions! > > > > > > > > >

Re: Samza 0.12.0 + synchronous KafkaProducer ?

st any > > failures in subsequent sends() from same process call. > > > > Thanks for being patient with my questions! > > > > > > > > From: Gaurav Agarwal > > Sent: Thursday, March 9, 2017 11:27:17 PM > > To: dev@samza.apache.org > > Cc: Muk

Re: Samza 0.12.0 + synchronous KafkaProducer ?

.. that seems to be >> problematic, as once checkpointed, there is no safety net against any >> failures in subsequent sends() from same process call. >> >> Thanks for being patient with my questions! >> >> >> >> Fr

Re: Samza 0.12.0 + synchronous KafkaProducer ?

or being patient with my questions! > > > > From: Gaurav Agarwal > Sent: Thursday, March 9, 2017 11:27:17 PM > To: dev@samza.apache.org > Cc: Mukul Gupta; Kshitij Gupta > Subject: Re: Samza 0.12.0 + synchronous KafkaProducer ? > > Hi J

Re: Samza 0.12.0 + synchronous KafkaProducer ?

subsequent sends() from same process call. Thanks for being patient with my questions! From: Gaurav Agarwal Sent: Thursday, March 9, 2017 11:27:17 PM To: dev@samza.apache.org Cc: Mukul Gupta; Kshitij Gupta Subject: Re: Samza 0.12.0 + synchronous KafkaProducer ? Hi

Re: Samza 0.12.0 + synchronous KafkaProducer ?

Hi Jagadish, please find reply inline: (it appears that there is no easy way today to guarantee ordered delivery of messages to Kafka from Samza without consuming the checkpointing flexibility). On Thu, Mar 9, 2017 at 11:01 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Gaurav, >

Re: Samza 0.12.0 + synchronous KafkaProducer ?

Hi Gaurav, >> process->process->->doWork()->checkpoint->process.. What does *doWork()* do? Does it actually iterate over accumulated in-memory state, and send messages to Kafka? *>> I found the configuration 'batch.size' which says that ''a batch size of zero will disable batching entirely"

Re: Samza 0.12.0 + synchronous KafkaProducer ?

Hi Jagadish, Thank you for very quick and detailed response. We have already set the task.commit.ms = -1 and are using the checkpointing mechanism to accumulate some work in memory in order to do it more efficiently in batches. So the flow is process->process->->doWork()->checkpoint->process.

Re: Samza 0.12.0 + synchronous KafkaProducer ?

Gaurav, I really appreciate your diligent walkthrough of the code base. Please find my replies inline. *>> I am trying to figure out, how to make our Samza task processing strictly ordered * By default, Samza offers you guaranteed in-order atleast-once processing out-of the box (same semantics a

Re: Samza 0.12.0 + synchronous KafkaProducer ?

(correcting recipient address) On Thu, Mar 9, 2017 at 10:39 AM, Gaurav Agarwal wrote: > Hi All, > > We are trying to upgrade to Kafka 0.12.0. In the process we noticed that > the Kafka 0.10.0 KafkaProducer client api does not provide any > configuration to send() the messages synchronously. One

Samza 0.12.0 + synchronous KafkaProducer ?

Hi All, We are trying to upgrade to Kafka 0.12.0. In the process we noticed that the Kafka 0.10.0 KafkaProducer client api does not provide any configuration to send() the messages synchronously. One needs to wait on the returned Future for synchronous guarantees. I am trying to figure out, how t