Thanks for the KIP, Satish.
I am trying to understand the problem we are looking to solve with this
KIP. When the leader is slow in processing fetch requests from the follower
(due to disk, GC, or other reasons), the primary problem is that it could
impact read and write latency and at times cause
Congratulations Konstantine! Well deserved!
On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen
wrote:
> Congratulations Konstantine!
>
> On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax wrote:
>
> > Congrats!
> >
> > On 6/21/21 12:57 PM, Raymond Ng wrote:
> > > Congrats Konstantine!
> > >
> > > /Ray
Dhruvil Shah created KAFKA-12520:
Summary: Producer state is needlessly rebuilt on startup
Key: KAFKA-12520
URL: https://issues.apache.org/jira/browse/KAFKA-12520
Project: Kafka
Issue Type
e go ahead and cherry-pick your fix onto the 2.8
> branch.
>
> Thanks!
> -John
>
> On Mon, 2021-03-01 at 09:36 -0800, Dhruvil Shah wrote:
> > Hi John,
> >
> > I would like to bring up
> https://issues.apache.org/jira/browse/KAFKA-12254
> > as a blocker
Hi John,
I would like to bring up https://issues.apache.org/jira/browse/KAFKA-12254
as a blocker candidate for 2.8.0. While this is not a regression, the
issue could lead to data loss in certain cases. The fix is trivial so it
may be worth bringing it into 2.8.0. Let me know what you think.
- Dhr
Dhruvil Shah created KAFKA-12254:
Summary: MirrorMaker 2.0 creates destination topic with default
configs
Key: KAFKA-12254
URL: https://issues.apache.org/jira/browse/KAFKA-12254
Project: Kafka
Dhruvil Shah created KAFKA-10518:
Summary: Consumer fetches could be inefficient when lags are
unbalanced
Key: KAFKA-10518
URL: https://issues.apache.org/jira/browse/KAFKA-10518
Project: Kafka
Dhruvil Shah created KAFKA-10517:
Summary: Inefficient consumer processing with fetch sessions
Key: KAFKA-10517
URL: https://issues.apache.org/jira/browse/KAFKA-10517
Project: Kafka
Issue
Hi Satish, Harsha,
Thanks for the KIP. Few questions below:
1. Could you describe how retention would work with this KIP and which
threads are responsible for driving this work? I believe there are 3 kinds
of retention processes we are looking at:
(a) Regular retention for data in tiered storag
Dhruvil Shah created KAFKA-9961:
---
Summary: Brokers may be left in an inconsistent state after
reassignment
Key: KAFKA-9961
URL: https://issues.apache.org/jira/browse/KAFKA-9961
Project: Kafka
Dhruvil Shah created KAFKA-9956:
---
Summary: Authorizer APIs may be invoked more than once for a given
request
Key: KAFKA-9956
URL: https://issues.apache.org/jira/browse/KAFKA-9956
Project: Kafka
Thanks for the KIP! +1 (non-binding)
On Tue, Apr 21, 2020 at 6:09 AM David Jacot wrote:
> Great KIP, thanks! +1 (non-binding)
>
> On Fri, Apr 17, 2020 at 8:56 PM Guozhang Wang wrote:
>
> > Thanks for the great KIP Kowshik, +1 (binding).
> >
> > On Fri, Apr 17, 2020 at 11:22 AM Jun Rao wrote:
>
Hi Kowshik,
Thanks for the KIP, this is exciting!
The KIP includes examples on how operators could use the command line
utility, etc. It would be great to add some high-level details on how the
upgrade workflow changes overall with the addition of feature versions.
- Dhruvil
On Wed, Apr 15, 202
Dhruvil Shah created KAFKA-9772:
---
Summary: Transactional offset commit fails with
IllegalStateException
Key: KAFKA-9772
URL: https://issues.apache.org/jira/browse/KAFKA-9772
Project: Kafka
environments, they could choose to disable the
feature altogether.
Let me know what you think. It would be good to hear what others think as
well.
Thanks,
Dhruvil
On Thu, Jan 16, 2020 at 3:24 AM Colin McCabe wrote:
> On Wed, Jan 15, 2020, at 03:54, Dhruvil Shah wrote:
> > Hi Colin,
&
give a time frame -- is it
> assumed to be immediate?
>
> best,
> Colin
>
>
> On Tue, Jan 14, 2020, at 05:56, Dhruvil Shah wrote:
> > If there are no more questions or concerns, I will start a vote thread
> > tomorrow.
> >
> > Thanks,
> > Dhruvil
> >
If there are no more questions or concerns, I will start a vote thread
tomorrow.
Thanks,
Dhruvil
On Mon, Jan 13, 2020 at 6:59 PM Dhruvil Shah wrote:
> Hi Nikhil,
>
> Thanks for looking at the KIP. The kind of race condition you mention is
> not possible as stray partition detec
tely solve this issue.
>
> Thanks
> Nikhil
>
> On 2020/01/06 04:30:20, Dhruvil Shah wrote:
> > Here is the link to the KIP:>
> >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker
> >
>
Here is the link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker
On Mon, Jan 6, 2020 at 9:59 AM Dhruvil Shah wrote:
> Hi all, I would like to kick off discussion for KIP-550 which proposes a
> mechanism to detect and
Hi all, I would like to kick off discussion for KIP-550 which proposes a
mechanism to detect and delete stray partitions on a broker. Suggestions
and feedback are welcome.
- Dhruvil
Dhruvil Shah created KAFKA-9307:
---
Summary: Transaction coordinator could be left in unknown state
after ZK session timeout
Key: KAFKA-9307
URL: https://issues.apache.org/jira/browse/KAFKA-9307
Project
Hi Satish,
Thanks for the KIP, those seems very useful. Could you elaborate on how
pending fetch requests are tracked?
Thanks,
Dhruvil
On Mon, Oct 28, 2019 at 9:43 PM Satish Duggana
wrote:
> Hi All,
> I wrote a short KIP about avoiding out-of-sync or offline partitions
> when follower fetch re
[
https://issues.apache.org/jira/browse/KAFKA-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhruvil Shah resolved KAFKA-8125.
-
Resolution: Duplicate
> Check for topic existence in CreateTopicsRequest prior to creat
Dhruvil Shah created KAFKA-8962:
---
Summary: KafkaAdminClient#describeTopics always goes through the
controller
Key: KAFKA-8962
URL: https://issues.apache.org/jira/browse/KAFKA-8962
Project: Kafka
This is exciting! +1 (non-binding)
- Dhruvil
On Thu, Sep 12, 2019 at 12:29 PM Bill Bejeck wrote:
> Thanks for the KIP!
>
> +1 (binding)
>
> -Bill
>
> On Thu, Sep 12, 2019 at 3:26 PM Ismael Juma wrote:
>
> > Thanks for the KIP, +1 (binding).
> >
> > Ismael
> >
> > On Mon, Sep 9, 2019, 8:28 AM C
Hi Justine,
Thanks for the KIP, this is great!
Could you add some more information about what deprecating the broker
configuration means? Would we log a warning in the logs when auto topic
creation is enabled on the broker, for example?
Thanks,
Dhruvil
On Thu, Jul 11, 2019 at 10:28 AM Justine O
Dhruvil Shah created KAFKA-8570:
---
Summary: Downconversion could fail when log contains out of order
message formats
Key: KAFKA-8570
URL: https://issues.apache.org/jira/browse/KAFKA-8570
Project: Kafka
Dhruvil Shah created KAFKA-8359:
---
Summary: Reconsider default for leader imbalance percentage
Key: KAFKA-8359
URL: https://issues.apache.org/jira/browse/KAFKA-8359
Project: Kafka
Issue Type
Thanks for the KIP! +1 (non-binding)
On Wed, May 8, 2019 at 10:23 PM Colin McCabe wrote:
> +1. Thanks, Aishwarya.
>
> Colin
>
> On Wed, May 8, 2019, at 17:50, Jason Gustafson wrote:
> > +1. Thanks!
> >
> > On Wed, May 8, 2019 at 4:30 PM Aishwarya Gune
> > wrote:
> >
> > > Hi All!
> > >
> > > I
Dhruvil Shah created KAFKA-8322:
---
Summary: Flaky test:
SslTransportLayerTest.testListenerConfigOverride
Key: KAFKA-8322
URL: https://issues.apache.org/jira/browse/KAFKA-8322
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhruvil Shah resolved KAFKA-8185.
-
Resolution: Not A Problem
This is not a typically expected scenario and would only happen when
Thanks for the KIP, Viktor! This is a useful addition. +1 overall.
Minor nits:
> I propose to add three gauge: DeadFetcherThreadCount for the fetcher
threads, log-cleaner-dead-thread-count for the log cleaner.
I think you meant two instead of three.
Also, would it make sense to name these metrics
The unit test itself does not seem to use too many files. What is the
output for `ulimit -n` on your system? Running `lsof` might also be helpful
to determine how many open files you have while Kafka is not running.
- Dhruvil
On Tue, Nov 27, 2018 at 9:20 AM lk gen wrote:
> When running ./gradle
Hi Dasun, seems like the screenshots were not attached. Could you please
open a Jira here: https://issues.apache.org/jira/projects/KAFKA
Thanks,
Dhruvil
On Tue, Oct 30, 2018 at 10:29 PM Dasun Nirmitha
wrote:
> Hello Guys
> I'm currently testing a Java Kafka producer application coded to retriev
Congratulations Colin!
On Tue, Sep 25, 2018 at 6:47 AM Stephane Maarek <
steph...@simplemachines.com.au> wrote:
> Congrats Colin !
>
> On Tue., 25 Sep. 2018, 3:33 pm Bill Bejeck, wrote:
>
> > Congrats Colin!
> >
> > On Tue, Sep 25, 2018 at 8:11 AM Manikumar
> > wrote:
> >
> > > Congrats Colin!
!
> On Sun, Sep 16, 2018 at 7:40 PM Matthias J. Sax
> wrote:
> >
> > +1 (binding)
> >
> > -Matthias
> >
> > On 9/14/18 4:57 PM, Ismael Juma wrote:
> > > Thanks for the KIP, +1 (binding).
> > >
> > > Ismael
> > &
Hi all,
I would like to start a vote on KIP-361.
Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
Thanks,
Dhruvil
>> Generally, I think positive configs (`allow` instead of `suppress`)
> >>>> are
> >>>>>>> easier to understand.
> >>>>>>>
> >>>>>>> Ismael
> >>>>>>>
> >>>>
Dhruvil Shah created KAFKA-7385:
---
Summary: Log compactor crashes when empty headers are retained
with idempotent / transaction producers
Key: KAFKA-7385
URL: https://issues.apache.org/jira/browse/KAFKA-7385
te.topics=true above to disallow auto
topic creation.
On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah wrote:
> To be clear, we will allow auto topic creation only when server config
> auto.create.topics.enable=true and consumer config
> allow.auto.create.topics=true; when either is fal
> >
> > or
> >
> > "allow.auto.topic.creation"
> >
> > w.r.t. suppress or allow, I don't have strong opinion either. It's just a
> > matter of choosing the proper default value.
> >
> > Cheers
> >
> >> On Wed,
ault as a first step (or warn
> every time it's used).
>
> Ismael
>
> On 21 Aug 2018 3:56 pm, "Dhruvil Shah" wrote:
>
> Hi,
>
> I would like to start discussion on KIP-361 that proposes we add a consumer
> configuration to disable auto topic creation.
" rather than "enable. The consumer
> > cannot enable auto topic creation because it is configured on the broker.
> > All it can do is prevent it from happening if it is enabled.
> >
> > -Jason
> >
> > On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah
>
Congratulations, Dong!
On Tue, Aug 21, 2018 at 4:38 PM Jason Gustafson wrote:
> Congrats!
>
> On Tue, Aug 21, 2018 at 10:03 AM, Ray Chiang wrote:
>
> > Congrats Dong!
> >
> > -Ray
> >
> >
> > On 8/21/18 9:33 AM, Becket Qin wrote:
> >
> >> Congrats, Dong!
> >>
> >> On Aug 21, 2018, at 11:03 PM,
Hi,
I would like to start discussion on KIP-361 that proposes we add a consumer
configuration to disable auto topic creation.
Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation
Suggestions and feedback are welco
Dhruvil Shah created KAFKA-7320:
---
Summary: Provide ability to disable auto topic creation in
KafkaConsumer
Key: KAFKA-7320
URL: https://issues.apache.org/jira/browse/KAFKA-7320
Project: Kafka
Thanks for the KIP, Stanislav! +1 (non-binding)
- Dhruvil
On Mon, Aug 13, 2018 at 9:39 AM Colin McCabe wrote:
> +1 (non-binding)
>
> best,
> Colin
>
> On Tue, Aug 7, 2018, at 04:19, Stanislav Kozlovski wrote:
> > Hey everybody,
> > I'm starting a vote on KIP-346
> > <
> https://cwiki.apache.org
For the cleaner thread specifically, I do not think respawning will help at
all because we are more than likely to run into the same issue again which
would end up crashing the cleaner. Retrying makes sense for transient
errors or when you believe some part of the system could have healed
itself, b
If we are looking into deprecating the empty group id, would it also make
sense to have the same character restriction for it as that for topic
names? We have stricter validation for topic names but none for group id
and transaction id. I think we should (eventually) make character
restriction the
Dhruvil Shah created KAFKA-7185:
---
Summary: getMatchingAcls throws StringIndexOutOfBoundsException
for empty resource name
Key: KAFKA-7185
URL: https://issues.apache.org/jira/browse/KAFKA-7185
Project
[
https://issues.apache.org/jira/browse/KAFKA-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhruvil Shah resolved KAFKA-7184.
-
Resolution: Not A Problem
> Kafka is going down with issue ERROR Failed to clean up log
+1 to what Jason said. We need a better long-term strategy for dealing with
corrupted log and index data, but the sanity checks we have do not
guarantee much in this regard.
For now, we could do away with these index sanity checks in my opinion. We
could handle the missing index case at startup. I
Dhruvil Shah created KAFKA-7076:
---
Summary: Broker startup could be inefficient when using old
message format
Key: KAFKA-7076
URL: https://issues.apache.org/jira/browse/KAFKA-7076
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhruvil Shah resolved KAFKA-6881.
-
Resolution: Not A Bug
Closing this JIRA because /tmp was being used as the log directory
Dhruvil Shah created KAFKA-7045:
---
Summary: Consumer may not be able to consume all messages when
down-conversion is required
Key: KAFKA-7045
URL: https://issues.apache.org/jira/browse/KAFKA-7045
Dhruvil Shah created KAFKA-7030:
---
Summary: Add configuration to disable message down-conversion
Key: KAFKA-7030
URL: https://issues.apache.org/jira/browse/KAFKA-7030
Project: Kafka
Issue Type
I agree that using `default.timeout.ms` could cause confusion since we
already have other timeout configurations in the consumer.
+1 for using `default.block.ms`.
Thanks,
Dhruvil
On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck wrote:
> Hi Jason,
>
> At first, I thought the same name between the p
Dhruvil Shah created KAFKA-6950:
---
Summary: Add mechanism to delay response to failed client
authentication
Key: KAFKA-6950
URL: https://issues.apache.org/jira/browse/KAFKA-6950
Project: Kafka
Mon, May 21, 2018 at 3:59 PM, Ismael Juma wrote:
> >
> > > Thanks for the KIP, +1 (binding).
> > >
> > > Ismael
> > >
> > > On Mon, May 21, 2018 at 7:52 AM Dhruvil Shah
> > wrote:
> > >
> > > > Hi,
> > > >
Dhruvil Shah created KAFKA-6927:
---
Summary: Broker uses significant amount of memory during
down-conversion
Key: KAFKA-6927
URL: https://issues.apache.org/jira/browse/KAFKA-6927
Project: Kafka
Hi,
I would like to start a vote on KIP-306 which proposes to add a
configuration to delay responses to failed authentication.
Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
Thanks,
Dhruvil
Hi,
I created a KIP that proposes we add a broker configuration to delay
responses to failed client authentication. This will help prevent DoS-like
situations because of a misconfigured application trying to connect with
incorrect or stale credentials over and over again.
Link to the KIP:
https:/
> Ismael
>
> On Wed, May 2, 2018 at 9:27 AM Dhruvil Shah wrote:
>
> > Hi all,
> >
> > I would like to start the vote on KIP-238: Efficient Memory Usage for
> > Down-Conversion.
> >
> > For reference, the link to the KIP is here:
> >
>
;enabled."
> >
> > -Jason
> >
> > On Wed, May 2, 2018 at 9:35 AM, Ted Yu wrote:
> >
> > > +1
> > >
> > > On Wed, May 2, 2018 at 9:27 AM, Dhruvil Shah
> > wrote:
> > >
> > > > Hi all,
> > > >
> &g
Hi all,
I would like to start the vote on KIP-238: Efficient Memory Usage for
Down-Conversion.
For reference, the link to the KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion
and the discussion thread is here:
https://www.mail-a
down-converting the entire partition upfront. Jun, let me know what you
think.
Any other suggestions / feedback are welcome.
Thanks,
Dhruvil
On Tue, Apr 17, 2018 at 4:21 PM, Dhruvil Shah wrote:
> Hi Jun,
>
> Yes, that is true. Ideally, we should be able to down-convert only the
> first
+1 from me as well. I was going to file a JIRA for this exact same problem.
:-)
On Thu, Apr 19, 2018 at 10:45 PM, Alex Dunayevsky
wrote:
> +1
>
>
> 4 votes total:
>
> 1 binding vote (Jason Gustafson)
>
> 3 non-binding votes (Moshe Blumberg, Ted Yu, Alex Dunayevsky)
>
>
> Can we consider the
first partition in the I/O thread (like we do today), and only delay
> down-conversion for subsequent partitions." It seems that we can further
> optimize this by only down-converting the first message set in the first
> partition in the request handling threads?
>
> Jun
>
ed size buffers, but your last note suggests that
> > you would use temporary buffers created for each partition. Do we need to
> > consider using a memory pool for these or do we think that the buffers
> will
> > be small enough to cope with lots of connection
t; will be a clear improvement over what we have now in any case, but just
> checking anyway.
>
> Regards,
>
> Rajini
>
> On Sat, Apr 7, 2018 at 12:29 AM, Dhruvil Shah
> wrote:
>
> > Hi Ted,
> >
> > Thanks for the comments.
> >
> >
> >
;
> bq. even if it exceeds fetch.max.bytes
>
> I did a brief search but didn't see the above config. Did you mean
> message.max.bytes
> ?
>
> bq. with possibility to grow if the allocation
>
> After the buffers grow, is there a way to trim them down if subsequent
>
I fixed the diagrams - let me know if you are still having trouble seeing
them.
Thanks,
Dhruvil
On Fri, Apr 6, 2018 at 3:05 PM, Ted Yu wrote:
> The two embedded diagrams seem broken.
>
> Can you double check ?
>
> Thanks
>
> On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil
Hi,
I created a KIP to help mitigate out of memory issues during
down-conversion. The KIP proposes introducing a configuration that can
prevent down-conversions altogether, and also describes a design for
efficient memory usage for down-conversion.
https://cwiki.apache.org/confluence/display/KAFK
73 matches
Mail list logo