Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Lucas Bradstreet]

Hi Harsha,

Is the problem you'd like addressed the following?

Assume 3 replicas, L and F1 and F2.

1. F1 and F2 are alive and sending fetch requests to L.
2. L starts encountering disk issues, any requests being processed by
the request handler threads become blocked.
3. L's zookeeper connection is still alive so it remains the leader
for the partition.
4. Given that F1 and F2 have not successfully fetched, L shrinks the
ISR to itself.

While KIP-501 may help prevent a shrink in partitions where a replica
fetch request has started processing, any fetch requests in the
request queue will have no effect. Generally when these slow/failing
disk issues occur, all of the request handler threads end up blocked
and requests queue up in the request queue. For example, all of the
request handler threads may end up stuck in
KafkaApis.handleProduceRequest handling produce requests, at which
point all of the replica fetcher fetch requests remain queued in the
request queue. If this happens, there will be no tracked fetch
requests to prevent a shrink.

Solving this shrinking issue is tricky. It would be better if L
resigns leadership when it enters a degraded state rather than
avoiding a shrink. If L is no longer the leader in this situation, it
will eventually become blocked fetching from the new leader and the
new leader will shrink the ISR, kicking out L.

Cheers,

Lucas

Build failed in Jenkins: kafka-2.4-jdk8 #138

[Apache Jenkins Server]

See 


Changes:

[vvcephei] KAFKA-9517: Fix default serdes with FK join (#8061)


--
[...truncated 5.50 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Harsha Ch]

Hi Jason & Jun,

                 Do you have any feedback on the KIP or is it ok take it to 
voting?. Its good to have this config in Kafka to address disk failure 
scenarios as described in the KIP.

Thanks,

Harsha

On Mon, Feb 10, 2020 at 5:10 PM, Brian Sang < bais...@yelp.com.invalid > wrote:

> 
> 
> 
> Hi,
> 
> 
> 
> Just wanted to bump this discussion, since it happened to us again at Yelp
> 
> 
> 
> 
> It's particularly nasty since it can happen right before a disk failure,
> so right as the leader for the partition becomes the only ISR, the leader
> becomes unrecoverable right after, forcing us to do an unclean leader
> election to resolve the situation. Having offline partitions due to a
> single failure is really annoying. I'm curious if others have experienced
> this as well, but weren't able to trace it to this specific error.
> 
> 
> 
> Best,
> Brian
> 
> 
> 
> On 2020/01/22 03:28:34, Satish Duggana < satish. duggana@ gmail. com (
> satish.dugg...@gmail.com ) > wrote:
> 
> 
>> 
>> 
>> Hi Jun,
>> Can you please review the KIP and let us know your comments?
>> 
>> 
>> 
>> If there are no comments/questions, we can start a vote thread.
>> 
>> 
>> 
>> It looks like Yelp folks also encountered the same issue as mentioned in
>> JIRA comment[1].
>> 
>> 
>>> 
 
 
 Flavien Raynaud added a comment - Yesterday
 
 
>>> 
>>> 
>> 
>> 
>> 
>> We've seen offline partitions happening for the same reason in one of our
>> clusters too, where only the broker leader for the offline partitions was
>> having disk issues. It looks like there has not been much progress/look on
>> the PR submitted since December 9th. Is there anything blocking this
>> change from moving forward?
>> 
>> 
>> 
>> 1. https:/ / issues. apache. org/ jira/ browse/ 
>> KAFKA-8733?focusedCommentId=17020083=com.
>> atlassian. jira. plugin. system. 
>> issuetabpanels%3Acomment-tabpanel#comment-17020083
>> (
>> https://issues.apache.org/jira/browse/KAFKA-8733?focusedCommentId=17020083=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17020083
>> )
>> 
>> 
>> 
>> Thanks,
>> Satish.
>> 
>> 
>> 
>> On Thu, Dec 5, 2019 at 10:38 AM Harsha Chintalapani < kafka@ harsha. io (
>> ka...@harsha.io ) > wrote:
>> 
>> 
>>> 
>>> 
>>> Hi Jason,
>>> As Satish said just increase replica max lag will not work in this case.
>>> Just before a disk dies the reads becomes really slow and its hard to
>>> estimate how much this is, as we noticed range is pretty wide. Overall it
>>> doesn't make sense to knock good replicas out of just because a leader is
>>> slower in processing reads or serving the fetch requests which may be due
>>> to disk issues in this case but could be other issues as well. I think
>>> this kip addresses in general all of these issues.
>>> Do you still have questions on the current approach if not we can take it
>>> vote.
>>> Thanks,
>>> Harsha
>>> 
>>> 
>>> 
>>> On Mon, Nov 18, 2019 at 7:05 PM, Satish Duggana < satish. duggana@ gmail. 
>>> com
>>> ( satish.dugg...@gmail.com ) > wrote:
>>> 
>>> 
 
 
 Hi Jason,
 Thanks for looking into the KIP. Apologies for my late reply. Increasing
 replica max lag to 30-45 secs did not help as we observed that a few fetch
 requests took more than 1-2 minutes. We do not want to increase further as
 it increases upper bound on commit latency. We have strict SLAs on some of
 the clusters on end to end(producer to consumer) latency. This proposal
 improves the availability of partitions when followers are trying their
 best to be insync even when leaders are slow in processing those requests.
 I have updated the KIP to have a single config for giving backward
 compatibility and I guess this config is more comprehensible than earlier.
 But I believe there is no need to have config because the suggested
 proposal in the KIP is an enhancement to the existing behavior. Please let
 me know your comments.
 
 
 
 Thanks,
 Satish.
 
 
 
 On Thu, Nov 14, 2019 at 10:57 AM Jason Gustafson < jason@ confluent. io (
 ja...@confluent.io ) > wrote:
 
 
 
 Hi Satish,
 
 
 
 Thanks for the KIP. I'm wondering how much of this problem can be
 addressed just by increasing the replication max lag? That was one of the
 purposes of KIP-537 (the default increased from 10s to 30s). Also, the new
 configurations seem quite low level. I think they will be hard for users
 to understand (even reading through a couple times I'm not sure I
 understand them fully). I think if there's a way to improve this behavior
 without requiring any new configurations, it would be much more
 attractive.
 
 
 
 Best,
 Jason
 
 
 
 On Wed, Nov 6, 2019 at 8:14 AM Satish Duggana < satish. duggana@ gmail. com
 ( satish.dugg...@gmail.com ) > wrote:
 
 
 
 Hi Dhruvil,
 Thanks for looking into the KIP.
 

[VOTE] KIP-568: Explicit rebalance triggering on the Consumer

[Sophie Blee-Goldman]

Hey all,

I'd like to start the voting on KIP-568. It proposes the new
Consumer#enforceRebalance API to facilitate triggering efficient rebalances.

For reference, here is the KIP link again:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-568%3A+Explicit+rebalance+triggering+on+the+Consumer

Thanks!
Sophie

Build failed in Jenkins: kafka-trunk-jdk8 #4221

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9423: Refine layout of configuration options on website and make

[github] KAFKA-9487: Follow-up PR of Kafka-9445 (#8033)

[manikumar] MINOR: Update schema field names in DescribeAcls Request/Response

[github] KAFKA-9505: Only loop over topics-to-validate in retries (#8039)

[github] KAFKA-9480: Fix bug that prevented to measure task-level process-rate

[github] KAFKA-9523: Migrate BranchedMultiLevelRepartitionConnectedTopologyTest

[github] KAFKA-9517: Fix default serdes with FK join (#8061)


--
[...truncated 2.83 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task 

Build failed in Jenkins: kafka-trunk-jdk11 #1146

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9505: Only loop over topics-to-validate in retries (#8039)

[github] KAFKA-9480: Fix bug that prevented to measure task-level process-rate

[github] KAFKA-9523: Migrate BranchedMultiLevelRepartitionConnectedTopologyTest

[github] KAFKA-9517: Fix default serdes with FK join (#8061)


--
[...truncated 2.85 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 

Re: [DISCUSS] KIP-568: Explicit rebalance triggering on the Consumer

[Sophie Blee-Goldman]

Thanks John. I took out the KafkaConsumer method and moved the javadocs
to the Consumer#enforceRebalance in the KIP -- hope you're happy :P

Also, I wanted to point out one minor change to the current proposal: make
this
a blocking call, which accepts a timeout and returns whether the rebalance
completed within the timeout. It will still reduce to a nonblocking call if
a "zero"
timeout is supplied. I've updated the KIP accordingly.

Let me know if there are any further concerns, else I'll call for a vote.

Cheers!
Sophie

On Mon, Feb 10, 2020 at 12:47 PM John Roesler  wrote:

> Thanks Sophie,
>
> Sorry I didn't respond. I think your new method name sounds good.
>
> Regarding the interface vs implementation, I agree it's confusing. It's
> always bothered me that the interface redirects you to an implementation
> JavaDocs, but never enough for me to stop what I'm doing to fix it.
> It's not a big deal either way, I just thought it was strange to propose a
> "public interface" change, but not in terms of the actual interface class.
>
> It _is_ true that KafkaConsumer is also part of the public API, but only
> really
> for the constructor. Any proposal to define a new "consumer client" API
> should be on the Consumer interface (which you said you plan to do anyway).
> I guess I brought it up because proposing an addition to Consumer implies
> it would be added to KafkaConsumer, but proposing an addition to
> KafkaConsumer does not necessarily imply it would also be added to
> Consumer. Does that make sense?
>
> Anyway, thanks for updating the KIP.
>
> Thanks,
> -John
>
>
> On Mon, Feb 10, 2020, at 14:38, Sophie Blee-Goldman wrote:
> > Since this doesn't seem too controversial, I'll probably call for a vote
> by
> > end of day.
> > If there any further comments/questions/concerns, please let me know!
> >
> > Thanks,
> > Sophie
> >
> > On Sat, Feb 8, 2020 at 12:19 AM Sophie Blee-Goldman  >
> > wrote:
> >
> > > Thanks for the feedback! That's a good point about trying to prevent
> users
> > > from
> > > thinking they should use this API during normal processing and
> clarifying
> > > when/why
> > > you might need it -- regardless of the method name, we should
> explicitly
> > > call this out
> > > in the javadocs.
> > >
> > > As for the method name, on reflection I agree that "rejoinGroup" does
> not
> > > seem to be
> > > appropriate. Of course that's what the consumer will actually be doing,
> > > but that's just an
> > > implementation detail -- the name should reflect what the API is doing,
> > > not how it does it
> > > (which can always change).
> > >
> > > How about "enforceRebalance"? This is stolen from the StreamThread
> method
> > > that does
> > > exactly this (by unsubscribing) so it seems to fit. I'll update the KIP
> > > with this unless anyone
> > > has another suggestion.
> > >
> > > Regarding the Consumer vs KafkaConsumer matter, I included the
> > > KafkaConsumer method
> > > because that's where all the javadocs redirect to in the Consumer
> > > interface. Also, FWIW
> > > I'm pretty sure KafkaConsumer is also part of the public API -- we
> would
> > > be adding a new
> > > method to both.
> > >
> > > On Fri, Feb 7, 2020 at 7:42 PM John Roesler 
> wrote:
> > >
> > >> Hi all,
> > >>
> > >> Thanks for the well motivated KIP, Sophie. I had some alternatives in
> > >> mind, which
> > >> I won't even bother to relate because I feel like the motivation made
> a
> > >> compelling
> > >> argument for the API as proposed.
> > >>
> > >> One very minor point you might as well fix is that the API change is
> > >> targeted at
> > >> KafkaConsumer (the implementation), but should be targeted at
> > >> Consumer (the interface).
> > >>
> > >> I agree with your discomfort about the name. Adding a "rejoin" method
> > >> seems strange
> > >> since there's no "join" method. Instead the way you join the group the
> > >> first time is just
> > >> by calling "subscribe". But "resubscribe" seems too indirect from what
> > >> we're really trying
> > >> to do, which is to trigger a rebalance by sending a new JoinGroup
> request.
> > >>
> > >> Another angle is that we don't want the method to sound like something
> > >> you should
> > >> be calling in normal circumstances, or people will be "tricked" into
> > >> calling it unnecessarily.
> > >>
> > >> So, I think "rejoinGroup" is fine, although a person _might_ be
> forgiven
> > >> for thinking they
> > >> need to call it periodically or something. Did you consider
> > >> "triggerRebalance", which
> > >> sounds pretty advanced-ish, and accurately describes what happens when
> > >> you call it?
> > >>
> > >> All in all, the KIP sounds good to me, and I'm in favor.
> > >>
> > >> Thanks,
> > >> -John
> > >>
> > >> On Fri, Feb 7, 2020, at 21:22, Anna McDonald wrote:
> > >> > This situation was discussed at length after a recent talk I gave.
> This
> > >> KIP
> > >> > would be a great step towards increased availability and in
> facilitating
> > >> > lightweight rebalances.

Build failed in Jenkins: kafka-trunk-jdk8 #4220

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8904: Improve producer's topic metadata fetching. (#7781)


--
[...truncated 2.40 MB...]
org.apache.kafka.streams.TopologyTest > 
sessionWindowNamedMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testShiftOffsetByWhenBetweenBeginningAndEndOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testShiftOffsetByWhenBetweenBeginningAndEndOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetToSpecificOffsetWhenAfterEndOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetToSpecificOffsetWhenAfterEndOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
shouldAcceptValidDateFormats STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
shouldAcceptValidDateFormats PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > shouldDeleteTopic STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > shouldDeleteTopic PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetToSpecificOffsetWhenBetweenBeginningAndEndOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetToSpecificOffsetWhenBetweenBeginningAndEndOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testShiftOffsetByWhenAfterEndOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testShiftOffsetByWhenAfterEndOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetToSpecificOffsetWhenBeforeBeginningOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetToSpecificOffsetWhenBeforeBeginningOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > shouldSeekToEndOffset 
STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > shouldSeekToEndOffset 
PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetUsingPlanWhenBetweenBeginningAndEndOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetUsingPlanWhenBetweenBeginningAndEndOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetUsingPlanWhenBeforeBeginningOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetUsingPlanWhenBeforeBeginningOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
shouldThrowOnInvalidDateFormat STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
shouldThrowOnInvalidDateFormat PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetUsingPlanWhenAfterEndOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testResetUsingPlanWhenAfterEndOffset PASSED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testShiftOffsetByWhenBeforeBeginningOffset STARTED

org.apache.kafka.streams.tools.StreamsResetterTest > 
testShiftOffsetByWhenBeforeBeginningOffset PASSED
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:873)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:484)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:693)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:658)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400)
at hudson.scm.SCM.poll(SCM.java:417)
at hudson.model.AbstractProject._poll(AbstractProject.java:1390)
at hudson.model.AbstractProject.poll(AbstractProject.java:1293)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:603)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:649)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
ERROR: Could not install GRADLE_4_10_3_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:873)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:484)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:693)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:658)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:400)
at hudson.scm.SCM.poll(SCM.java:417)
   

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Brian Sang]

Hi,

Just wanted to bump this discussion, since it happened to us again at Yelp  

It's particularly nasty since it can happen right before a disk failure, so 
right as the leader for the partition becomes the only ISR, the leader becomes 
unrecoverable right after, forcing us to do an unclean leader election to 
resolve the situation. Having offline partitions due to a single failure is 
really annoying. I'm curious if others have experienced this as well, but 
weren't able to trace it to this specific error.

Best,
Brian

On 2020/01/22 03:28:34, Satish Duggana  wrote: 
> Hi Jun,
> Can you please review the KIP and let us know your comments?
> 
> If there are no comments/questions, we can start a vote thread.
> 
> It looks like Yelp folks also encountered the same issue as mentioned
> in JIRA comment[1].
> 
> >> Flavien Raynaud added a comment - Yesterday
> We've seen offline partitions happening for the same reason in one of
> our clusters too, where only the broker leader for the offline
> partitions was having disk issues. It looks like there has not been
> much progress/look on the PR submitted since December 9th. Is there
> anything blocking this change from moving forward?
> 
> 1. 
> https://issues.apache.org/jira/browse/KAFKA-8733?focusedCommentId=17020083=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17020083
> 
> Thanks,
> Satish.
> 
> 
> On Thu, Dec 5, 2019 at 10:38 AM Harsha Chintalapani  wrote:
> >
> > Hi Jason,
> >  As Satish said just increase replica max lag will not work in this
> > case. Just before a disk dies the reads becomes really slow and its hard to
> > estimate how much this is, as we noticed range is pretty wide. Overall it
> > doesn't make sense to knock good replicas out of just because a leader is
> > slower in processing reads or serving the fetch requests which may be due
> > to disk issues in this case but could be other issues as well. I think this
> > kip addresses in general all of these issues.
> >  Do you still have questions on the current approach if not we can
> > take it vote.
> > Thanks,
> > Harsha
> >
> >
> > On Mon, Nov 18, 2019 at 7:05 PM, Satish Duggana 
> > wrote:
> >
> > > Hi Jason,
> > > Thanks for looking into the KIP. Apologies for my late reply. Increasing
> > > replica max lag to 30-45 secs did not help as we observed that a few fetch
> > > requests took more than 1-2 minutes. We do not want to increase further as
> > > it increases upper bound on commit latency. We have strict SLAs on some of
> > > the clusters on end to end(producer to consumer) latency. This proposal
> > > improves the availability of partitions when followers are trying their
> > > best to be insync even when leaders are slow in processing those requests.
> > > I have updated the KIP to have a single config for giving backward
> > > compatibility and I guess this config is more comprehensible than earlier.
> > > But I believe there is no need to have config because the suggested
> > > proposal in the KIP is an enhancement to the existing behavior. Please let
> > > me know your comments.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Thu, Nov 14, 2019 at 10:57 AM Jason Gustafson 
> > > wrote:
> > >
> > > Hi Satish,
> > >
> > > Thanks for the KIP. I'm wondering how much of this problem can be
> > > addressed just by increasing the replication max lag? That was one of the
> > > purposes of KIP-537 (the default increased from 10s to 30s). Also, the new
> > > configurations seem quite low level. I think they will be hard for users 
> > > to
> > > understand (even reading through a couple times I'm not sure I understand
> > > them fully). I think if there's a way to improve this behavior without
> > > requiring any new configurations, it would be much more attractive.
> > >
> > > Best,
> > > Jason
> > >
> > > On Wed, Nov 6, 2019 at 8:14 AM Satish Duggana 
> > > wrote:
> > >
> > > Hi Dhruvil,
> > > Thanks for looking into the KIP.
> > >
> > > 10. I have an initial sketch of the KIP-500 in commit[a] which discusses
> > > tracking the pending fetch requests. Tracking is not done in
> > > Partition#readRecords because if it takes longer in reading any of the
> > > partitions then we do not want any of the replicas of this fetch request 
> > > to
> > > go out of sync.
> > >
> > > 11. I think `Replica` class should be thread-safe to handle the remote
> > > scenario of concurrent requests running for a follower replica. Or I may 
> > > be
> > > missing something here. This is a separate issue from KIP-500. I will file
> > > a separate JIRA to discuss that issue.
> > >
> > > a -
> > > https://github.com/satishd/kafka/commit/
> > > c69b525abe8f6aad5059236076a003cdec4c4eb7
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Tue, Oct 29, 2019 at 10:57 AM Dhruvil Shah 
> > > wrote:
> > >
> > > Hi Satish,
> > >
> > > Thanks for the KIP, those seems very useful. Could you elaborate on how
> > > pending fetch requests are tracked?
> > >
> > > Thanks,
> > > 

[jira] [Created] (KAFKA-9536) Integration tests for KIP-558

[Konstantine Karantasis (Jira)]

Konstantine Karantasis created KAFKA-9536:
-

 Summary: Integration tests for KIP-558
 Key: KAFKA-9536
 URL: https://issues.apache.org/jira/browse/KAFKA-9536
 Project: Kafka
  Issue Type: Test
Reporter: Konstantine Karantasis
Assignee: Konstantine Karantasis


Extend testing coverage for 
[KIP-558|https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect]
 with integration tests and additional unit tests. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk11 #1145

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-2.5-jdk8 #12

[Apache Jenkins Server]

See 


Changes:

[manikumar] MINOR: Update schema field names in DescribeAcls Request/Response


--
[...truncated 2.85 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

[jira] [Created] (KAFKA-9535) Metadata not updated when consumer encounters leader epoch related failures

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9535:
--

 Summary: Metadata not updated when consumer encounters leader 
epoch related failures
 Key: KAFKA-9535
 URL: https://issues.apache.org/jira/browse/KAFKA-9535
 Project: Kafka
  Issue Type: Bug
Reporter: Boyang Chen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-523 Add KStream#toTable to the Streams DSL

[Sang wn Lee]

Hello All 

I added scala API to KIP
Please check.
Thank you! 

On 2019/10/24 13:19:40, aishwarya kumar  wrote: 
> Hello All,
> 
> After concluding discussions for this KIP, I would like to go forward with
> the voting process.
> 
> Jira Ticket: https://issues.apache.org/jira/browse/KAFKA-7658
> KIP :
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL
> 
> Thank you,
> Aishwarya
> 

[jira] [Resolved] (KAFKA-9523) Reduce flakiness of BranchedMultiLevelRepartitionConnectedTopologyTest

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9523.
--
Fix Version/s: 2.5.0
   Resolution: Fixed

> Reduce flakiness of BranchedMultiLevelRepartitionConnectedTopologyTest
> --
>
> Key: KAFKA-9523
> URL: https://issues.apache.org/jira/browse/KAFKA-9523
> Project: Kafka
>  Issue Type: Test
>  Components: streams, unit tests
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
> Fix For: 2.5.0
>
>
> KAFKA-9335 introduces an integration test to verify the topology builder 
> itself could survive from building a complex topology. This test gets flaky 
> some time for stream client to broker connection, so we should consider 
> making it less flaky by either converting to a unit test or just focus on 
> making the test logic more robust.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9505) InternalTopicManager may falls into infinite loop with partially created topics

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9505.
--
Fix Version/s: 2.5.0
   Resolution: Fixed

> InternalTopicManager may falls into infinite loop with partially created 
> topics
> ---
>
> Key: KAFKA-9505
> URL: https://issues.apache.org/jira/browse/KAFKA-9505
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 2.5.0
>
>
> In {{InternalTopicManager#validateTopics(topicsNotReady, topics)}}, the 
> topics map (second) does not change while the first topicsNotReady may change 
> if some topics have been validated while others do not, however inside that 
> function we still loop of the second map which may never completes then.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-568: Explicit rebalance triggering on the Consumer

[John Roesler]

Thanks Sophie,

Sorry I didn't respond. I think your new method name sounds good.

Regarding the interface vs implementation, I agree it's confusing. It's
always bothered me that the interface redirects you to an implementation
JavaDocs, but never enough for me to stop what I'm doing to fix it.
It's not a big deal either way, I just thought it was strange to propose a 
"public interface" change, but not in terms of the actual interface class.

It _is_ true that KafkaConsumer is also part of the public API, but only really
for the constructor. Any proposal to define a new "consumer client" API
should be on the Consumer interface (which you said you plan to do anyway).
I guess I brought it up because proposing an addition to Consumer implies
it would be added to KafkaConsumer, but proposing an addition to 
KafkaConsumer does not necessarily imply it would also be added to 
Consumer. Does that make sense?

Anyway, thanks for updating the KIP.

Thanks,
-John


On Mon, Feb 10, 2020, at 14:38, Sophie Blee-Goldman wrote:
> Since this doesn't seem too controversial, I'll probably call for a vote by
> end of day.
> If there any further comments/questions/concerns, please let me know!
> 
> Thanks,
> Sophie
> 
> On Sat, Feb 8, 2020 at 12:19 AM Sophie Blee-Goldman 
> wrote:
> 
> > Thanks for the feedback! That's a good point about trying to prevent users
> > from
> > thinking they should use this API during normal processing and clarifying
> > when/why
> > you might need it -- regardless of the method name, we should explicitly
> > call this out
> > in the javadocs.
> >
> > As for the method name, on reflection I agree that "rejoinGroup" does not
> > seem to be
> > appropriate. Of course that's what the consumer will actually be doing,
> > but that's just an
> > implementation detail -- the name should reflect what the API is doing,
> > not how it does it
> > (which can always change).
> >
> > How about "enforceRebalance"? This is stolen from the StreamThread method
> > that does
> > exactly this (by unsubscribing) so it seems to fit. I'll update the KIP
> > with this unless anyone
> > has another suggestion.
> >
> > Regarding the Consumer vs KafkaConsumer matter, I included the
> > KafkaConsumer method
> > because that's where all the javadocs redirect to in the Consumer
> > interface. Also, FWIW
> > I'm pretty sure KafkaConsumer is also part of the public API -- we would
> > be adding a new
> > method to both.
> >
> > On Fri, Feb 7, 2020 at 7:42 PM John Roesler  wrote:
> >
> >> Hi all,
> >>
> >> Thanks for the well motivated KIP, Sophie. I had some alternatives in
> >> mind, which
> >> I won't even bother to relate because I feel like the motivation made a
> >> compelling
> >> argument for the API as proposed.
> >>
> >> One very minor point you might as well fix is that the API change is
> >> targeted at
> >> KafkaConsumer (the implementation), but should be targeted at
> >> Consumer (the interface).
> >>
> >> I agree with your discomfort about the name. Adding a "rejoin" method
> >> seems strange
> >> since there's no "join" method. Instead the way you join the group the
> >> first time is just
> >> by calling "subscribe". But "resubscribe" seems too indirect from what
> >> we're really trying
> >> to do, which is to trigger a rebalance by sending a new JoinGroup request.
> >>
> >> Another angle is that we don't want the method to sound like something
> >> you should
> >> be calling in normal circumstances, or people will be "tricked" into
> >> calling it unnecessarily.
> >>
> >> So, I think "rejoinGroup" is fine, although a person _might_ be forgiven
> >> for thinking they
> >> need to call it periodically or something. Did you consider
> >> "triggerRebalance", which
> >> sounds pretty advanced-ish, and accurately describes what happens when
> >> you call it?
> >>
> >> All in all, the KIP sounds good to me, and I'm in favor.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Fri, Feb 7, 2020, at 21:22, Anna McDonald wrote:
> >> > This situation was discussed at length after a recent talk I gave. This
> >> KIP
> >> > would be a great step towards increased availability and in facilitating
> >> > lightweight rebalances.
> >> >
> >> > anna
> >> >
> >> >
> >> >
> >> > On Fri, Feb 7, 2020, 9:38 PM Sophie Blee-Goldman 
> >> > wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > In light of some recent and upcoming rebalancing and availability
> >> > > improvements, it seems we have a need for explicitly triggering a
> >> consumer
> >> > > group rebalance. Therefore I'd like to propose adding a new
> >> > > rejoinGroup()method
> >> > > to the Consumer client (better method name suggestions are very
> >> welcome).
> >> > >
> >> > > Please take a look at the KIP and let me know what you think!
> >> > >
> >> > > KIP document:
> >> > >
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-568%3A+Explicit+rebalance+triggering+on+the+Consumer
> >> > >
> >> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-9525
> >> > >
> 

Re: [DISCUSS] KIP-568: Explicit rebalance triggering on the Consumer

[Sophie Blee-Goldman]

Since this doesn't seem too controversial, I'll probably call for a vote by
end of day.
If there any further comments/questions/concerns, please let me know!

Thanks,
Sophie

On Sat, Feb 8, 2020 at 12:19 AM Sophie Blee-Goldman 
wrote:

> Thanks for the feedback! That's a good point about trying to prevent users
> from
> thinking they should use this API during normal processing and clarifying
> when/why
> you might need it -- regardless of the method name, we should explicitly
> call this out
> in the javadocs.
>
> As for the method name, on reflection I agree that "rejoinGroup" does not
> seem to be
> appropriate. Of course that's what the consumer will actually be doing,
> but that's just an
> implementation detail -- the name should reflect what the API is doing,
> not how it does it
> (which can always change).
>
> How about "enforceRebalance"? This is stolen from the StreamThread method
> that does
> exactly this (by unsubscribing) so it seems to fit. I'll update the KIP
> with this unless anyone
> has another suggestion.
>
> Regarding the Consumer vs KafkaConsumer matter, I included the
> KafkaConsumer method
> because that's where all the javadocs redirect to in the Consumer
> interface. Also, FWIW
> I'm pretty sure KafkaConsumer is also part of the public API -- we would
> be adding a new
> method to both.
>
> On Fri, Feb 7, 2020 at 7:42 PM John Roesler  wrote:
>
>> Hi all,
>>
>> Thanks for the well motivated KIP, Sophie. I had some alternatives in
>> mind, which
>> I won't even bother to relate because I feel like the motivation made a
>> compelling
>> argument for the API as proposed.
>>
>> One very minor point you might as well fix is that the API change is
>> targeted at
>> KafkaConsumer (the implementation), but should be targeted at
>> Consumer (the interface).
>>
>> I agree with your discomfort about the name. Adding a "rejoin" method
>> seems strange
>> since there's no "join" method. Instead the way you join the group the
>> first time is just
>> by calling "subscribe". But "resubscribe" seems too indirect from what
>> we're really trying
>> to do, which is to trigger a rebalance by sending a new JoinGroup request.
>>
>> Another angle is that we don't want the method to sound like something
>> you should
>> be calling in normal circumstances, or people will be "tricked" into
>> calling it unnecessarily.
>>
>> So, I think "rejoinGroup" is fine, although a person _might_ be forgiven
>> for thinking they
>> need to call it periodically or something. Did you consider
>> "triggerRebalance", which
>> sounds pretty advanced-ish, and accurately describes what happens when
>> you call it?
>>
>> All in all, the KIP sounds good to me, and I'm in favor.
>>
>> Thanks,
>> -John
>>
>> On Fri, Feb 7, 2020, at 21:22, Anna McDonald wrote:
>> > This situation was discussed at length after a recent talk I gave. This
>> KIP
>> > would be a great step towards increased availability and in facilitating
>> > lightweight rebalances.
>> >
>> > anna
>> >
>> >
>> >
>> > On Fri, Feb 7, 2020, 9:38 PM Sophie Blee-Goldman 
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > In light of some recent and upcoming rebalancing and availability
>> > > improvements, it seems we have a need for explicitly triggering a
>> consumer
>> > > group rebalance. Therefore I'd like to propose adding a new
>> > > rejoinGroup()method
>> > > to the Consumer client (better method name suggestions are very
>> welcome).
>> > >
>> > > Please take a look at the KIP and let me know what you think!
>> > >
>> > > KIP document:
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-568%3A+Explicit+rebalance+triggering+on+the+Consumer
>> > >
>> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-9525
>> > >
>> > > Cheers,
>> > > Sophie
>> > >
>> >
>>
>

Build failed in Jenkins: kafka-2.5-jdk8 #11

[Apache Jenkins Server]

See 

Changes:


--
[...truncated 2.86 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] 

[jira] [Created] (KAFKA-9534) Potential inconsistent result for listTopic talking to older brokers

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9534:
--

 Summary: Potential inconsistent result for listTopic talking to 
older brokers
 Key: KAFKA-9534
 URL: https://issues.apache.org/jira/browse/KAFKA-9534
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 2.5.0
Reporter: Boyang Chen


Not sure this is indeed a bug, just making it under track.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1144

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9423: Refine layout of configuration options on website and make

[github] KAFKA-9487: Follow-up PR of Kafka-9445 (#8033)


--
[...truncated 2.85 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task 

[jira] [Resolved] (KAFKA-9423) Refine layout of configuration options on website and make individual settings directly linkable

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-9423.
---
Fix Version/s: 2.6.0
   Resolution: Fixed

> Refine layout of configuration options on website and make individual 
> settings directly linkable
> 
>
> Key: KAFKA-9423
> URL: https://issues.apache.org/jira/browse/KAFKA-9423
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Sönke Liebau
>Assignee: Sönke Liebau
>Priority: Trivial
> Fix For: 2.6.0
>
> Attachments: option1.png, option2.png, option3.png, option4.png
>
>
> KAFKA-8474 changed the layout of configuration options on the website from a 
> table which over time ran out of horizontal space to a list.
> This vastly improved readability but is not yet ideal. Further discussion was 
> had in the [PR|https://github.com/apache/kafka/pull/6870] for KAFKA-8474.
> This ticket is to move that discussion to a separate thread and make it more 
> visible to other people and to give subsequent PRs a home.
> Currently proposed options are attached to this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9533) ValueTransform forwards `null` values

[Michael Viamari (Jira)]

Michael Viamari created KAFKA-9533:
--

 Summary: ValueTransform forwards `null` values
 Key: KAFKA-9533
 URL: https://issues.apache.org/jira/browse/KAFKA-9533
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.4.0
Reporter: Michael Viamari


According to the documentation for `KStream#transformValues`, nulls returned 
from `ValueTransformer#transform` are not forwarded. (see 
[KStream#transformValues|https://kafka.apache.org/24/javadoc/org/apache/kafka/streams/kstream/KStream.html#transformValues-org.apache.kafka.streams.kstream.ValueTransformerSupplier-java.lang.String...-]

However, this does not appear to be the case. In 
`KStreamTransformValuesProcessor#transform` the result of the transform is 
forwarded directly.
{code:java}
 @Override
 public void process(final K key, final V value) {
 context.forward(key, valueTransformer.transform(key, value));
 }
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Kafka 2.5.0 release

[David Arthur]

Just a friendly reminder that this Wednesday, February 12th, is the code
freeze for the 2.5.0 release. After this time we will only accept blocker
bugs onto the release branch.

Thanks!
David

On Fri, Jan 31, 2020 at 5:13 PM David Arthur  wrote:

> Thanks! I've updated the list.
>
> On Thu, Jan 30, 2020 at 5:48 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
>> Hi David,
>>
>> thanks for driving the release.
>>
>> Please also remove KIP-158 from the list of KIPs that you plan to include
>> in 2.5
>> KIP-158 has been accepted, but the implementation is not yet final. It
>> will be included in the release that follows 2.5.
>>
>> Regards,
>> Konstantine
>>
>> On 1/30/20, Matthias J. Sax  wrote:
>> > Hi David,
>> >
>> > the following KIP from the list did not make it:
>> >
>> >  - KIP-216 (no PR yet)
>> >  - KIP-399 (no PR yet)
>> >  - KIP-401 (PR not merged yet)
>> >
>> >
>> > KIP-444 should be included as we did make progress, but it is still not
>> > fully implement and we need to finish in in 2.6 release.
>> >
>> > KIP-447 is partially implemented in 2.5 (ie, broker and
>> > consumer/producer changes -- the Kafka Streams parts slip)
>> >
>> >
>> > -Matthias
>> >
>> >
>> > On 1/29/20 9:05 AM, David Arthur wrote:
>> >> Hey everyone, just a quick update on the 2.5 release.
>> >>
>> >> I have updated the list of planned KIPs on the release wiki page
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=143428858
>> .
>> >> If I have missed anything, or there are KIPs included in this list
>> which
>> >> should *not* be included in 2.5, please let me know.
>> >>
>> >> Based on the release schedule, the feature freeze is today, Jan 29th.
>> Any
>> >> major feature work that is not already complete will need to push out
>> to
>> >> 2.6. I will work on cutting the release branch during the day tomorrow
>> >> (Jan
>> >> 30th).
>> >>
>> >> If you have any questions, please feel free to reach out to me directly
>> >> or
>> >> in this thread.
>> >>
>> >> Thanks!
>> >> David
>> >>
>> >> On Mon, Jan 13, 2020 at 1:35 PM Colin McCabe 
>> wrote:
>> >>
>> >>> +1.  Thanks for volunteering, David.
>> >>>
>> >>> best,
>> >>> Colin
>> >>>
>> >>> On Fri, Dec 20, 2019, at 10:59, David Arthur wrote:
>>  Greetings!
>> 
>>  I'd like to volunteer to be release manager for the next time-based
>> >>> feature
>>  release which will be 2.5. If there are no objections, I'll send out
>>  the
>>  release plan in the next few days.
>> 
>>  Thanks,
>>  David Arthur
>> 
>> >>>
>> >>
>> >>
>> >
>> >
>>
>
>
> --
> David Arthur
>


-- 
David Arthur

[jira] [Resolved] (KAFKA-8623) KafkaProducer possible deadlock when sending to different topics

[Brian Byrne (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Byrne resolved KAFKA-8623.

Fix Version/s: 2.3.0
   Resolution: Fixed

This appears to be due to an issue concerning the handling of consecutive 
metadata updates in clients, where the first update could effectively clear the 
request for the second because no version/instance which request was 
outstanding was maintained. This was fixed in PR 
[6621|https://github.com/apache/kafka/pull/6221] (see item 3), which is 
available in the 2.3.0 release.

> KafkaProducer possible deadlock when sending to different topics
> 
>
> Key: KAFKA-8623
> URL: https://issues.apache.org/jira/browse/KAFKA-8623
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 2.2.1
>Reporter: Alexander Bagiev
>Assignee: Kun Song
>Priority: Critical
> Fix For: 2.3.0
>
>
> Project with bug reproduction: [https://github.com/abagiev/kafka-producer-bug]
> It was found that sending two messages in two different topics in a row 
> results in hanging of KafkaProducer for 60s and the following exception:
> {noformat}
> org.springframework.kafka.core.KafkaProducerException: Failed to send; nested 
> exception is org.apache.kafka.common.errors.TimeoutException: Failed to 
> update metadata after 6 ms.
>   at 
> org.springframework.kafka.core.KafkaTemplate.lambda$buildCallback$0(KafkaTemplate.java:405)
>  ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:877)
>  ~[kafka-clients-2.0.1.jar:na]
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803) 
> ~[kafka-clients-2.0.1.jar:na]
>   at 
> org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:444)
>  ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
>   at 
> org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:381) 
> ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
>   at 
> org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:193) 
> ~[spring-kafka-2.2.7.RELEASE.jar:2.2.7.RELEASE]
> ...
> {noformat}
> It looks like KafkaProducer requests two times for meta information for each 
> topic and hangs just before second request due to some deadlock. When 60s 
> pass TimeoutException is thrown and meta information is requested/received 
> immediately (but after exception has been already thrown).
> The issue in the example project is reproduced every time; and the use case 
> is trivial.
>  This is a critical bug for us.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk11 #1143

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8904: Improve producer's topic metadata fetching. (#7781)


--
[...truncated 2.85 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

Re: [DISCUSS] KIP-566: Add rebalance callbacks to ConsumerInterceptor

[Thomas Becker]

Bumping this again for visibility. If no one has any comments, maybe I'll just 
start the VOTE thread?


On Wed, 2020-01-29 at 22:24 +, Thomas Becker wrote:

[EXTERNAL EMAIL] Attention: This email was sent from outside TiVo. DO NOT CLICK 
any links or attachments unless you expected them.





Bumping this, hoping to get some additional feedback.



From: M. Manna mailto:manme...@gmail.com>>

Sent: Thursday, January 23, 2020 4:37 PM

To: dev@kafka.apache.org 
mailto:dev@kafka.apache.org>>

Subject: Re: [DISCUSS] KIP-566: Add rebalance callbacks to ConsumerInterceptor


[EXTERNAL EMAIL] Attention: This email was sent from outside TiVo. DO NOT CLICK 
any links or attachments unless you expected them.





Hey Thomas,


On Thu, 23 Jan 2020 at 21:17, Thomas Becker 
mailto:thomas.bec...@tivo.com>> wrote:


Hi folks,

I'd like to open the discussion for KIP-566: Add rebalance callbacks to

ConsumerInterceptor. We've been looking to implement some custom metrics

via ConsumerInterceptor, and not knowing when partition ownership changes

is a significant impediment. I'd appreciate your thoughts.



https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-566%253A%2BAdd%2Brebalance%2Bcallbacks%2Bto%2BConsumerInterceptordata=02%7C01%7CThomas.Becker%40tivo.com%7C660a1b4a8ba94d668c7708d7a50a10d3%7Cd05b7c6912014c0db45d7f1dcc227e4d%7C1%7C0%7C637159334967045569sdata=RYUcrnvDWady9%2FAnLSvc2vvLlchIbj6w0og8Vxe1KN4%3Dreserved=0




 I had a quick read through the KIP. I don't see any obvious issues.

Sounds like a simple improvement. Perhaps, you could add your thoughts

about RebalanceListener API in the future e.g. when to unify the

functionality. If implemented, we can simply use one implementation for

both things.


I would be interested to hear others' comments about this.


Thanks,







This email and any attachments may contain confidential and privileged

material for the sole use of the intended recipient. Any review, copying,

or distribution of this email (or any attachments) by others is prohibited.

If you are not the intended recipient, please contact the sender

immediately and permanently delete this email and any attachments. No

employee or agent of TiVo is authorized to conclude any binding agreement

on behalf of TiVo by email. Binding agreements with TiVo may only be made

by a signed written agreement.






This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
is authorized to conclude any binding agreement on behalf of TiVo by email. 
Binding agreements with TiVo may only be made by a signed written agreement.


--
[cid:9c5eb4be88341189d58cf171e402c865e96c80a8.camel@tivo.com] Tommy Becker
Principal Engineer
Personalized Content Discovery
O +1 919.460.4747
tivo.com



This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
is authorized to conclude any binding agreement on behalf of TiVo by email. 
Binding agreements with TiVo may only be made by a signed written agreement.

[jira] [Created] (KAFKA-9532) Deleting the consumer group programatically using RESTFul API

[Rakshith Mamidala (Jira)]

Rakshith  Mamidala created KAFKA-9532:
-

 Summary: Deleting the consumer group programatically using RESTFul 
API
 Key: KAFKA-9532
 URL: https://issues.apache.org/jira/browse/KAFKA-9532
 Project: Kafka
  Issue Type: Wish
  Components: clients
Affects Versions: 2.4.0
Reporter: Rakshith  Mamidala
 Fix For: 2.4.0


As a requirement in project, instead of listening the messages and consuming / 
storing message data into database, we are creating the consumer groups run 
time per user (to avoid thread safe issue) and using consumer.poll and 
consumer.seekToBeginning and once read all the messages we are closing the 
connection, unsubscribing consumer group. 

 

Whats happening in Kafka is, the consumer groups moved from active state to 
DEAD state but not getting removed / deleted, in Kafka Tools it shows all the 
consumers even if those are DEAD.

 

*What we want:*
 # How to remove / delete the consumer groups programatically.
 # Is there any REST Endpoint / command line / script to delete the consumer 
groups? What are those.
 # What impact the DEAD consumer groups can creates in terms of production 
environment.?

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9531) java.net.UnknownHostException loop on VM rolling update using CNAME

[Rui Abreu (Jira)]

Rui Abreu created KAFKA-9531:


 Summary: java.net.UnknownHostException loop on VM rolling update 
using CNAME
 Key: KAFKA-9531
 URL: https://issues.apache.org/jira/browse/KAFKA-9531
 Project: Kafka
  Issue Type: Bug
  Components: clients, controller, producer 
Affects Versions: 2.4.0
Reporter: Rui Abreu


Hello,

 

My cluster setup in based on VMs behind DNS CNAME .

Example:  node.internal is a CNAME to either nodeA.internal or nodeB.internal

Since kafka-client 1.2.1,  it has been observed that sometimes Kafka clients 
get stuck on a loop with the exception:
Example after nodeB.internal is replaced with nodeA.internal 

 
{code:java}
2020-02-10T12:11:28.181Z o.a.k.c.NetworkClient [WARN]- [Consumer 
clientId=consumer-6, groupId=consumer.group] Error connecting to node 
nodeB.internal:9092 (id: 2 rack: null)
java.net.UnknownHostException: nodeB.internal:9092
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) 
~[?:1.8.0_222]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) 
~[?:1.8.0_222]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) 
~[?:1.8.0_222]
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104) 
~[stormjar.jar:?]
at 
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943) 
~[stormjar.jar:?]
at 
org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68) 
~[stormjar.jar:?]
at 
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005)
 ~[stormjar.jar:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537) 
~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:366)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
 ~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1220) 
~[stormjar.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159) 
~[stormjar.jar:?]
at 
org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:365) 
~[stormjar.jar:?]
at 
org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:294) 
~[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__10715$fn__10730$fn__10761.invoke(executor.clj:649)
 ~[storm-core-1.1.3.jar:1.1.3]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
~[storm-core-1.1.3.jar:1.1.3]
at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
{code}
 

The time it spends in the loop is arbitrary, but it seems the client 
effectively stops while this is happening.

This error contrasts with instances where the client is able to recover on its 
own after a few seconds:


{code:java}
2020-02-08T01:15:37.390Z o.a.k.c.c.i.AbstractCoordinator [INFO] - [Consumer 
clientId=consumer-7, groupId=consumer-group] Group coordinator 
nodeA.internal:9092 (id: 2147483645 rack: null) is unavailable or invalid, will 
attempt rediscovery
 
2020-02-08T01:15:37.885Z o.a.k.c.c.i.AbstractCoordinator [INFO] - [Consumer 
clientId=consumer-7, groupId=consumer-group] Discovered group coordinator 
nodeB.internal:9092 (id: 2147483646 rack: null)

2020-02-08T01:15:37.885Z o.a.k.c.ClusterConnectionStates [INFO] - [Consumer 
clientId=consumer-7, groupId=consumer-group] Hostname for node 2147483646 
changed from nodeA.internal to nodeB.internal
{code}


   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

[David Jacot]

Hi Michael,

Thank you for the updated KIP. I have few comments/questions:

1. We already have the "--state" option in the command line tool which can
be used
with "--describe" and we will have "--states" which can be used with
"--list". I feel this
is going to be confusing for users. I wonder if we could combine both cases
to reduce
the confusion but I am not sure it would be better. What do you think?

2. Regarding the output of the command line when "--states" is used, I
wonder if it
wouldn't be better to use a proper table with a header. We could use only
when
filters such as "--states" are used.

Best,
David

On Thu, Feb 6, 2020 at 10:44 PM Colin McCabe  wrote:

> Hi Mickael,
>
> Can you please specify what the result is when a newer client tries to use
> this on an older broker?  Does that result in an
> UnsupportedVersionException?
>
> I would prefer an Optional in the Java API so that “show all groups” can
> be EMPTY.
>
> best,
> Colin
>
>
> On Mon, Jan 27, 2020, at 07:53, Mickael Maison wrote:
> > Hi David,
> >
> > Did that answer your questions? or do you have any further feedback?
> >
> > Thanks
> >
> > On Thu, Jan 16, 2020 at 4:11 PM Mickael Maison 
> wrote:
> > >
> > > Hi David,
> > >
> > > Thanks for taking a look.
> > > 1) Yes, updated
> > >
> > > 2) I had not considered that but indeed this would be useful if the
> > > request contained multiple states and would avoid doing another call.
> > > The ListGroups response already includes the group ProtocolType, so I
> > > guess we could add the State as well. The response will still be
> > > significantly smaller than DescribeGroups. With this change, one thing
> > > to note is that having Describe on the Cluster resource will allow
> > > retrieving the state of all groups. Currently retrieving the state of
> > > a group requires Describe on the Group.
> > >
> > > 3) Yes if ListGroups response includes the state, it makes sense to
> > > expose it via the command line tool and the AdminClient. With
> > > ConsumerGroupCommand, to avoid compatibility issues we can only print
> > > states when the --states flag is specified.
> > >
> > > I've updated the KIP accordingly.
> > >
> > > On Mon, Jan 13, 2020 at 12:20 PM David Jacot 
> wrote:
> > > >
> > > > Hi Michael,
> > > >
> > > > Please, excuse me for my late feedback. I've got a few
> questions/comments
> > > > while reviewing the KIP.
> > > >
> > > > 1. I would suggest to clearly state in the documentation of the
> state field
> > > > that omitting it or providing an empty list means "all".
> > > >
> > > > 2. Have you considered including the state in the response? The API
> allows
> > > > to search for multiple states so it could be
> > > > convenient to have the state in the response to let the user
> differentiate
> > > > the groups.
> > > >
> > > > 3. If 2. makes sense, I would suggest to also include it in the
> information
> > > > printed out by the ConsumerGroupCommand tool. Putting
> > > > myself in the shoes of an operator, I would like to see the state of
> each
> > > > group if I select specific states. Perhaps we could
> > > > use a table instead of the simple list used today. What do you think?
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Mon, Nov 11, 2019 at 12:40 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > If there's no more feedback, I'll open a vote in the next few days.
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Fri, Nov 1, 2019 at 4:27 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > Thanks for taking a look at the KIP!
> > > > > > You are right, even if we serialize the field as a String, we
> should
> > > > > > use ConsumerGroupState in the API.
> > > > > > As suggested, I've also updated the API so a list of states is
> specified.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > >
> > > > > > On Tue, Oct 22, 2019 at 10:03 AM Tom Bentley <
> tbent...@redhat.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > Thanks for the KIP.
> > > > > > >
> > > > > > > The use of String to represent the desired state in the API
> seems less
> > > > > > > typesafe than would be ideal. Is there a reason not to use the
> existing
> > > > > > > ConsumerGroupState enum (even if the state is serialized as a
> String)?
> > > > > > >
> > > > > > > While you say that the list-of-names result from
> listConsumerGroups is
> > > > > a
> > > > > > > reason not to support supplying a set of desired states I
> don't find
> > > > > that
> > > > > > > argument entirely convincing. Sure, if the results are going
> to be
> > > > > shown to
> > > > > > > a user then it would be ambiguous and multiple queries would be
> > > > > needed. But
> > > > > > > it seems quite possible that the returned list of groups will
> > > > > immediately
> > > > > > > be used in a 

Re: [DISCUSS] KIP-552: Add interface to handle unused config

[Patrik Kleindl]

Hi John
Starting an empty streams instance

final String bootstrapServers = "broker0:9092";
Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "configDemo");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
bootstrapServers);
StreamsBuilder builder = new StreamsBuilder();
final KafkaStreams streams = new KafkaStreams(builder.build(),
streamsConfiguration);
streams.start();

results in:

stream-thread [configDemo-bcaf82b4-324d-4956-a2a8-1dea0a8e3a2e-StreamThread-1]
Creating consumer client
ConsumerConfig values:
...
stream-thread 
[configDemo-bcaf82b4-324d-4956-a2a8-1dea0a8e3a2e-StreamThread-1-consumer]
Cooperative rebalancing enabled now
The configuration 'admin.retries' was supplied but isn't a known config.
The configuration 'admin.retry.backoff.ms' was supplied but isn't a
known config.
Kafka version: 2.4.0

when the normal consumer is created, but not for admin client /
producer / restore consumer.

StreamsConfig seems to include this on purpose:

final AdminClientConfig adminClientDefaultConfig = new
AdminClientConfig(getClientPropsWithPrefix(ADMIN_CLIENT_PREFIX,
AdminClientConfig.configNames()));
consumerProps.put(adminClientPrefix(AdminClientConfig.RETRIES_CONFIG),
adminClientDefaultConfig.getInt(AdminClientConfig.RETRIES_CONFIG));
consumerProps.put(adminClientPrefix(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG),
adminClientDefaultConfig.getLong(AdminClientConfig.RETRY_BACKOFF_MS_CONFIG));

If I add

streamsConfiguration.put(StreamsConfig.restoreConsumerPrefix(ConsumerConfig.RECEIVE_BUFFER_CONFIG),
65536);
streamsConfiguration.put(StreamsConfig.mainConsumerPrefix(ConsumerConfig.MAX_POLL_RECORDS_CONFIG),
100);

then the warnings

The configuration 'main.consumer.max.poll.records' was supplied but
isn't a known config.
The configuration 'restore.consumer.receive.buffer.bytes' was supplied
but isn't a known config.

are shown for all clients, not only the last consumer.

Streams provides these prefixes so maybe they are not handled
correctly regarding the log message.

Maybe this helps to pinpoint the source of this in KS at least

best regards

Patrik


On Sat, 8 Feb 2020 at 05:11, John Roesler  wrote:

> Looking at where the log message comes from:
> org.apache.kafka.common.config.AbstractConfig#logUnused
> it seems like maybe the warning just happens when you pass
> extra configs to a client that it has no knowledge of (and therefore
> doesn't "use").
>
> I'm now suspicious if Streams is actually sending extra configs to the
> clients, although it seems like we _don't_ see these warnings in other
> cases.
>
> Maybe some of the folks who actually see these messages can try to pinpoint
> where exactly the rogue configs are coming from?
>
> I might have overlooked a message at some point, but it wasn't clear to
> me that we were talking about warnings that were actually caused by
> Streams.
> I thought the unknown configs were something user-specified.
>
> Thanks,
> -John
>
> On Fri, Feb 7, 2020, at 13:10, Gwen Shapira wrote:
> > Ah, got it! I am indeed curious why they do this :)
> >
> > Maybe John can shed more light. But if we can't find a better fix,
> > perhaps the nice thing to do is really a separate logger, so users who
> > are not worried about shooting themselves in the foot can make those
> > warnings go away.
> >
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
> > On Fri, Feb 07, 2020 at 4:13 AM, Patrik Kleindl < pklei...@gmail.com >
> wrote:
> >
> > >
> > >
> > >
> > > Hi Gwen
> > >
> > >
> > >
> > > Kafka Streams is not a third party library and produces a lot of these
> > > warnings, e.g.
> > >
> > >
> > >
> > > *The configuration 'main.consumer.max.poll.records' was supplied but
> isn't
> > > a known config.*
> > > *The configuration 'admin.retries' was supplied but isn't a known
> config.*
> > > and various others if you try to fine-tune the restoration consumer or
> > > inject parameters for state stores.
> > > This results in a lot of false positives and only makes new people
> worried
> > > and then ignore the warnings altogether.
> > >
> > >
> > >
> > > Unless this is taken care of at least the Kafka Streams users will
> > > probably be better off having this on debug level.
> > >
> > >
> > >
> > > Best regards
> > >
> > >
> > >
> > > Patrik
> > >
> > >
> > >
> > > On Thu, 6 Feb 2020 at 16:55, Gwen Shapira < gwen@ confluent. io (
> > > g...@confluent.io ) > wrote:
> > >
> > >
> > >>
> > >>
> > >> INFO is the default log level, and while it looks less "alarming" than
> > >> WARN, users will still see it and in my experience, they will worry
> that
> > >> something is wrong anyway. Or if INFO isn't the default, users won't
> see
> > >> it, so it is no different from debug and we are left with no way of
> > >> warning users that they misconfigured something.
> > >>
> > >>
> > >>
> > >> The point is that "known configs" exist in Kafka as a