Re: [DISCUSS] KIP-405 + KAFKA-7739 - Implementation of Tiered Storage Integration with Azure Storage

Hi Israel, Linkedin is interested in evaluating KIP-405 for HDFS and S3 in the short term and Azure Blob Storage in the long run. You may already know that Linkedin is migrating to Azure . We think that Blobs will provide us with the

[jira] [Created] (KAFKA-6701) synchronize Log modification between delete cleanup and async delete

Sumant Tambe created KAFKA-6701: --- Summary: synchronize Log modification between delete cleanup and async delete Key: KAFKA-6701 URL: https://issues.apache.org/jira/browse/KAFKA-6701 Project: Kafka

[jira] [Created] (KAFKA-5886) Implement KIP-91

Sumant Tambe created KAFKA-5886: --- Summary: Implement KIP-91 Key: KAFKA-5886 URL: https://issues.apache.org/jira/browse/KAFKA-5886 Project: Kafka Issue Type: Improvement Components

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

On Mon, Sep 11, 2017 at 10:06 AM, Sumant Tambe wrote: > > > @Jun, Until we make idempotent producer the default (kip-185), this kip > is > > sensitive to retries. I.e., we expire batches either delivery.timeout.ms > > passes or all retries are exhausted, whichever comes fir

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

ct of such changes can remain in the > compatibility > > > section. Also, I agree about keeping your "reordering" text although it > > > seems like the wiki wasn't updated to match what you posted in the > > > discussion thread. > > >

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

agree about keeping your "reordering" text although it > seems like the wiki wasn't updated to match what you posted in the > discussion thread. > > Ismael > > On Fri, Sep 8, 2017 at 6:06 AM, Sumant Tambe wrote: > > > Hi all, > > > > I wou

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

mpatibility section, your note on > 'max.in.flight.requests.per.connection == 5' resulting in out of order > delivery is true irrespective of these changes. As such, I don't think it > should be mentioned in the context of this KIP. > > Thanks, > Apurva > > On Thu, Sep 7, 2

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

koff, we could also consider that. So, I'm not sure > > retries > > > has much use apart from compatibility and the retries=0 case (for now). > > > > > > Ismael > > > > > > On Wed, Sep 6, 2017 at 11:14 PM, Jun Rao wrote: > > > >

[VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

Hi all, I would like to open the vote for KIP-91: https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer Thank you all for your input on the kip so far. Regards, Sumant

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

> > > > For retries, probably we can just default it to MAX_INT? > > > > Thanks, > > > > Jun > > > > > > On Wed, Sep 6, 2017 at 10:26 AM, Sumant Tambe wrote: > > > > > 120 seconds default sounds good to me. Throwing ConfigExcep

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

ng). > > 3. We should mention that we will not cancel in flight requests until the > request timeout even though we'll expire the batch early if needed. > > I think we should start the vote tomorrow so that we have a chance of > hitting the KIP freeze for 1.0.0. > >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

rown, there is no guarantee whether the > > > > messages > > > > > are delivered or not. > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

is in retry because we expired an in-flight request > > before > > > > it > > > > > > hits request.timeout.ms. > > > > > > > > > > > > The difference between 2 and 3 is that in case 3 likely the > broker > > > has > > > > > &

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

way, the producer has more chance for batching. > The > >>> implication is that a batch could be closed longer than linger.ms. > >>> > >>> Now, on your concern about not having a precise way to control delay in > >>> the > >>> accu

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

ltiple times and new issues keep coming back. Ideally, > this time, we want to have a solution that covers all cases, even though > that requires a bit more work. > > Thanks, > > Jun > > > On Fri, Aug 11, 2017 at 12:30 PM, Sumant Tambe wrote: > > > Hi Jun, >

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

n this formula? Users of future.get can use this timeout value. Thoughts? Regards, Sumant On 11 August 2017 at 07:50, Sumant Tambe wrote: > > Thanks for the KIP. Nice documentation on all current issues with the >> timeout. > > For the KIP writeup, all credit goes to J

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

ly not too bad. The benefit is that this is more intuitive to the > end user. > > Does that sound reasonable to you? > > Thanks, > > Jun > > > On Wed, Aug 9, 2017 at 10:03 PM, Sumant Tambe wrote: > > > On Wed, Aug 9, 2017 at 1:28 PM Apurva Mehta wrote: > &g

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

On Wed, Aug 9, 2017 at 1:28 PM Apurva Mehta wrote: > > > There seems to be no relationship with cluster metadata availability or > > > staleness. Expiry is just based on the time since the batch has been > > ready. > > > Please correct me if I am wrong. > > > > > > > I was not very specific about

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Responses inline. > > However, one thing which has not come out of the JIRA discussion is the > > > actual use cases for batch expiry. > > > > There are two usecases I can think of for batch expiry mechanism > > irrespective of how we try to bound the time (batch.expiry.ms or > > max.message.deliv

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

n extends TimeoutException and includes some context, such as TopicPartition, broker-id, an app may provide differentiated service based on a topic name or availability zone of a broker-id. KIP-91 does not propose anything like that. It's a very niche usecase though. Regards, Sumant > &

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Just a quick comment. Can you list the alternatives > mentioned in the JIRA discussion in the rejected alternatives section? > > -Jason > > On Thu, Aug 3, 2017 at 3:09 PM, Sumant Tambe wrote: > > > Hi all, > > > > KIP-91 [1] is another attempt to get better c

[DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Hi all, KIP-91 [1] is another attempt to get better control on producer side timeouts. In essence we're proposing a new config named batch.expiry.ms that will cause batches in the accumulator to expire after the configured timeout. Recently, the discussion on KAFKA-5621 [2] has shed new light on

[jira] [Created] (KAFKA-4395) KafkaConfig and LogConfig should not have static initialization order dependencies

Sumant Tambe created KAFKA-4395: --- Summary: KafkaConfig and LogConfig should not have static initialization order dependencies Key: KAFKA-4395 URL: https://issues.apache.org/jira/browse/KAFKA-4395

[jira] [Updated] (KAFKA-4395) KafkaConfig and LogConfig should not have static initialization order dependencies

[ https://issues.apache.org/jira/browse/KAFKA-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4395: Affects Version/s: 0.10.0.1 > KafkaConfig and LogConfig should not have static initialization or

[jira] [Assigned] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe reassigned KAFKA-4089: --- Assignee: Sumant Tambe (was: Dong Lin) > KafkaProducer raises Batch Expired except

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when pro

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when pro

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when pro

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The basic idea of batch expiration is that we don't expire batches when pro

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumant Tambe updated KAFKA-4089: Description: The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}}) ejects

[jira] [Created] (KAFKA-4089) KafkaProducer raises Batch Expired exception

Sumant Tambe created KAFKA-4089: --- Summary: KafkaProducer raises Batch Expired exception Key: KAFKA-4089 URL: https://issues.apache.org/jira/browse/KAFKA-4089 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1911) Log deletion on stopping replicas should be async

[ https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392877#comment-15392877 ] Sumant Tambe commented on KAFKA-1911: - Hi [~mgharat], I would like to take your p