Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
vote thread created.
On Mon, Oct 21, 2019 at 9:58 AM Colin McCabe wrote:
>
> Hi Radai,
>
> It seems reasonable to me.
>
> best,
> Colin
>
>
> On Mon, Oct 21, 2019, at 09:52, radai wrote:
> > yet another bump.
> >
> > can we please have a vote if there are no objections ?
> >
> > On Wed, Sep 25, 2019 at 1:28 PM radai wrote:
> > >
> > > bump.
> > >
> > > so if no more concerns, can we move to a vote on this ?
> > >
> > > On Fri, Sep 13, 2019 at 10:05 AM radai wrote:
> > > >
> > > > we have a lot of processes that need a time-bounded checkpoint logic.
> > > >
> > > > the standard use case is some consume-process-produce logic of the
> > > > following form:
> > > >
> > > > while (alive) {
> > > >data = consumer.poll()
> > > >output = process(data)
> > > >producer.send(output)
> > > >
> > > >if (System.millis() > nextCheckpoint) {
> > > > nextCheckpoint = System.millis() + 5 minutes;
> > > > if (!producer.flush(30 seconds) || !consumer.commitSync(30
> > > > seconds)) {
> > > > //unable to checkpoint within timeout, die (or at least raise
> > > > some alarm)
> > > > }
> > > >}
> > > > }
> > > >
> > > > an unbounded flush could cause the consumer to be considered dead and
> > > > send the whole app into rebalance storms, so we'd really love to be
> > > > able to put a time bound on it.
> > > >
> > > > im also fine with rethrowing InterruptedException
> > > >
> > > > On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe wrote:
> > > > >
> > > > > Hi Radai,
> > > > >
> > > > > Thanks for the KIP. Sounds interesting. I assume that if an
> > > > > InterruptedException were caught, that would be rethrown, rather than
> > > > > returning false? It might be good to specify that. Can you give an
> > > > > example of how this would be used?
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Thu, Sep 12, 2019, at 15:26, radai wrote:
> > > > > > bump.
> > > > > >
> > > > > > if no one has any comments on this can we initiate a vote?
> > > > > >
> > > > > > On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I would like to start discussion on KIP-514 that proposes we add a
> > > > > > > bounded flush() API to producer.
> > > > > > >
> > > > > > > Link to the KIP:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > > > > > >
> > > > > > > Suggestions and feedback are welcome!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Kun
> > > > > >
> >
Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
Hi Radai,
It seems reasonable to me.
best,
Colin
On Mon, Oct 21, 2019, at 09:52, radai wrote:
> yet another bump.
>
> can we please have a vote if there are no objections ?
>
> On Wed, Sep 25, 2019 at 1:28 PM radai wrote:
> >
> > bump.
> >
> > so if no more concerns, can we move to a vote on this ?
> >
> > On Fri, Sep 13, 2019 at 10:05 AM radai wrote:
> > >
> > > we have a lot of processes that need a time-bounded checkpoint logic.
> > >
> > > the standard use case is some consume-process-produce logic of the
> > > following form:
> > >
> > > while (alive) {
> > >data = consumer.poll()
> > >output = process(data)
> > >producer.send(output)
> > >
> > >if (System.millis() > nextCheckpoint) {
> > > nextCheckpoint = System.millis() + 5 minutes;
> > > if (!producer.flush(30 seconds) || !consumer.commitSync(30
> > > seconds)) {
> > > //unable to checkpoint within timeout, die (or at least raise
> > > some alarm)
> > > }
> > >}
> > > }
> > >
> > > an unbounded flush could cause the consumer to be considered dead and
> > > send the whole app into rebalance storms, so we'd really love to be
> > > able to put a time bound on it.
> > >
> > > im also fine with rethrowing InterruptedException
> > >
> > > On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe wrote:
> > > >
> > > > Hi Radai,
> > > >
> > > > Thanks for the KIP. Sounds interesting. I assume that if an
> > > > InterruptedException were caught, that would be rethrown, rather than
> > > > returning false? It might be good to specify that. Can you give an
> > > > example of how this would be used?
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Thu, Sep 12, 2019, at 15:26, radai wrote:
> > > > > bump.
> > > > >
> > > > > if no one has any comments on this can we initiate a vote?
> > > > >
> > > > > On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I would like to start discussion on KIP-514 that proposes we add a
> > > > > > bounded flush() API to producer.
> > > > > >
> > > > > > Link to the KIP:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > > > > >
> > > > > > Suggestions and feedback are welcome!
> > > > > >
> > > > > > Thanks,
> > > > > > Kun
> > > > >
>
Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
yet another bump.
can we please have a vote if there are no objections ?
On Wed, Sep 25, 2019 at 1:28 PM radai wrote:
>
> bump.
>
> so if no more concerns, can we move to a vote on this ?
>
> On Fri, Sep 13, 2019 at 10:05 AM radai wrote:
> >
> > we have a lot of processes that need a time-bounded checkpoint logic.
> >
> > the standard use case is some consume-process-produce logic of the
> > following form:
> >
> > while (alive) {
> >data = consumer.poll()
> >output = process(data)
> >producer.send(output)
> >
> >if (System.millis() > nextCheckpoint) {
> > nextCheckpoint = System.millis() + 5 minutes;
> > if (!producer.flush(30 seconds) || !consumer.commitSync(30 seconds)) {
> > //unable to checkpoint within timeout, die (or at least raise
> > some alarm)
> > }
> >}
> > }
> >
> > an unbounded flush could cause the consumer to be considered dead and
> > send the whole app into rebalance storms, so we'd really love to be
> > able to put a time bound on it.
> >
> > im also fine with rethrowing InterruptedException
> >
> > On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe wrote:
> > >
> > > Hi Radai,
> > >
> > > Thanks for the KIP. Sounds interesting. I assume that if an
> > > InterruptedException were caught, that would be rethrown, rather than
> > > returning false? It might be good to specify that. Can you give an
> > > example of how this would be used?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Sep 12, 2019, at 15:26, radai wrote:
> > > > bump.
> > > >
> > > > if no one has any comments on this can we initiate a vote?
> > > >
> > > > On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I would like to start discussion on KIP-514 that proposes we add a
> > > > > bounded flush() API to producer.
> > > > >
> > > > > Link to the KIP:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > > > >
> > > > > Suggestions and feedback are welcome!
> > > > >
> > > > > Thanks,
> > > > > Kun
> > > >
Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
bump.
so if no more concerns, can we move to a vote on this ?
On Fri, Sep 13, 2019 at 10:05 AM radai wrote:
>
> we have a lot of processes that need a time-bounded checkpoint logic.
>
> the standard use case is some consume-process-produce logic of the
> following form:
>
> while (alive) {
>data = consumer.poll()
>output = process(data)
>producer.send(output)
>
>if (System.millis() > nextCheckpoint) {
> nextCheckpoint = System.millis() + 5 minutes;
> if (!producer.flush(30 seconds) || !consumer.commitSync(30 seconds)) {
> //unable to checkpoint within timeout, die (or at least raise
> some alarm)
> }
>}
> }
>
> an unbounded flush could cause the consumer to be considered dead and
> send the whole app into rebalance storms, so we'd really love to be
> able to put a time bound on it.
>
> im also fine with rethrowing InterruptedException
>
> On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe wrote:
> >
> > Hi Radai,
> >
> > Thanks for the KIP. Sounds interesting. I assume that if an
> > InterruptedException were caught, that would be rethrown, rather than
> > returning false? It might be good to specify that. Can you give an
> > example of how this would be used?
> >
> > best,
> > Colin
> >
> >
> > On Thu, Sep 12, 2019, at 15:26, radai wrote:
> > > bump.
> > >
> > > if no one has any comments on this can we initiate a vote?
> > >
> > > On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote:
> > > >
> > > > Hi,
> > > >
> > > > I would like to start discussion on KIP-514 that proposes we add a
> > > > bounded flush() API to producer.
> > > >
> > > > Link to the KIP:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > > >
> > > > Suggestions and feedback are welcome!
> > > >
> > > > Thanks,
> > > > Kun
> > >
Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
we have a lot of processes that need a time-bounded checkpoint logic.
the standard use case is some consume-process-produce logic of the
following form:
while (alive) {
data = consumer.poll()
output = process(data)
producer.send(output)
if (System.millis() > nextCheckpoint) {
nextCheckpoint = System.millis() + 5 minutes;
if (!producer.flush(30 seconds) || !consumer.commitSync(30 seconds)) {
//unable to checkpoint within timeout, die (or at least raise
some alarm)
}
}
}
an unbounded flush could cause the consumer to be considered dead and
send the whole app into rebalance storms, so we'd really love to be
able to put a time bound on it.
im also fine with rethrowing InterruptedException
On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe wrote:
>
> Hi Radai,
>
> Thanks for the KIP. Sounds interesting. I assume that if an
> InterruptedException were caught, that would be rethrown, rather than
> returning false? It might be good to specify that. Can you give an example
> of how this would be used?
>
> best,
> Colin
>
>
> On Thu, Sep 12, 2019, at 15:26, radai wrote:
> > bump.
> >
> > if no one has any comments on this can we initiate a vote?
> >
> > On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote:
> > >
> > > Hi,
> > >
> > > I would like to start discussion on KIP-514 that proposes we add a
> > > bounded flush() API to producer.
> > >
> > > Link to the KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > >
> > > Suggestions and feedback are welcome!
> > >
> > > Thanks,
> > > Kun
> >
Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
Hi Radai, Thanks for the KIP. Sounds interesting. I assume that if an InterruptedException were caught, that would be rethrown, rather than returning false? It might be good to specify that. Can you give an example of how this would be used? best, Colin On Thu, Sep 12, 2019, at 15:26, radai wrote: > bump. > > if no one has any comments on this can we initiate a vote? > > On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote: > > > > Hi, > > > > I would like to start discussion on KIP-514 that proposes we add a > > bounded flush() API to producer. > > > > Link to the KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer > > > > Suggestions and feedback are welcome! > > > > Thanks, > > Kun >
Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer
bump. if no one has any comments on this can we initiate a vote? On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote: > > Hi, > > I would like to start discussion on KIP-514 that proposes we add a > bounded flush() API to producer. > > Link to the KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer > > Suggestions and feedback are welcome! > > Thanks, > Kun
