Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Matthias J. Sax]

Currently we, don't cleanup dead threads, but the KIP proposes to change
this:

> Stream threads that are in state DEAD will be removed from the stream threads 
> of a Kafka Streams client.


-Matthias

On 9/8/20 2:37 PM, Sophie Blee-Goldman wrote:
> Ah, I forgot about localThreadsMetadata(). In that. case I agree, there's
> no reason
> to introduce a new method when we can get both the names and number of all
> running threads from this.
> 
> I assume that we would update localThreadsMetadata to only return currently
> alive threads as part of this KIP -- at a quick glance, it seems like we
> don't do
> any pruning of dead threads at the moment
> 
> On Tue, Sep 8, 2020 at 1:58 PM Matthias J. Sax  wrote:
> 
>> I am not sure if we need a new method? There is already
>> `localThreadsMetadata()`. What do we gain by adding a new one?
>>
>> Returning the thread's name (as `Optional`) for both add() and
>> remove() is fine with me.
>>
>>
>> -Matthias
>>
>> On 9/8/20 12:58 PM, Sophie Blee-Goldman wrote:
>>> Sorry Bruno, I think I missed the end of your message with the
>>> numberOfAliveStreamThreads()
>>> proposal. I agree, that would be better than the alternatives I listed.
>>> That said:
>>>
 They rather suggest that the method returns a list of handles to the
>>> stream threads.
>>>
>>> I hadn't thought of that originally, but now that you mention it, this
>>> might be a good idea.
>>> I don't think we should return actual handles on the threads, but maybe a
>>> list of the thread
>>> names rather than a single number of currently alive threads.
>>>
>>> Since we seem to think it would be difficult if not impossible to keep
>>> track of the number
>>> of running stream threads, we should apply the same reasoning to the
>> names
>>> and not
>>> assume the user can/will keep track of every thread returned by
>>> addStreamThread() or
>>> removeStreamThread(). Users should generally take any required action
>>> immediately
>>> after adding/removing the thread -- eg deregistering the thread metrics
>> --
>>> but it might
>>> still be useful to provide a convenience method listing all of the
>> current
>>> threads
>>>
>>> And of course you could still get the number of threads easily by
>> invoking
>>> size() on the
>>> returned list (or ordered set?).
>>>
>>> On Tue, Sep 8, 2020 at 12:16 PM Bruno Cadonna 
>> wrote:
>>>
 Thank you again for the feedback Sophie!

 As I tried to point out in my previous e-mail, removing a stream thread
 from a Kafka Streams client that does not have alive stream threads is
 nothing exceptional for the client per se. However, it can become
 exceptional within the context of the user. For example, if users want
 to remove a stream thread from a client without alive stream threads
 because one if their metrics say so, then this is exceptional in the
 context of that user metric not in the context of the Kafka Streams
 client. In that case, users should throw an exception and handle it.

 Regarding returning null, I do not like to return null because from a
 development point of view there is no distinction between returning null
 because we have a bug in the code or returning null because there are no
 alive stream threads. Additionally, Optional makes it more
 explicit that the result could also be empty.

 Thank you for the alternative method names! However, with the names you
 propose it is not immediately clear that the method returns an amount of
 stream threads. They rather suggest that the method returns a list of
 handles to the stream threads. I chose to use "aliveStreamThreads" to be
 consistent with the client-level metric "alive-stream-threads" which
 reports the same number of stream threads that
 numberOfAliveStreamThreads() should report. If others also think that
 the proposed name in the KIP is too clumsy, I am open to rename it,
>> though.

 Best,
 Bruno


 On 08.09.20 20:12, Sophie Blee-Goldman wrote:
>> it's never a good sign when the discussion moves into the vote thread
>
> Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads
 strikes
> again.
> Thanks for redirecting me Bruno
>
> I suppose it's unfair to expect the callers to keep perfect track of
>> the
> current
>   number of stream threads, but it also seems like you shouldn't be
 calling
> removeStreamThread() when there are no threads left. Either you're just
> haphazardly removing threads and could unintentionally slip into a
>> state
 of
> no
> running threads without realizing it, or more realistically, you're
> carefully
> removing threads based on some metric(s) that convey whether the system
 is
> over or under-provisioned. If your metrics say you're over-provisioned
 but
> there's
> not one thread running, well, that certainly sounds exceptional to me.
>> Or
> you 

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

[Matthias J. Sax]

+1 (binding)

On 9/8/20 2:49 PM, Jason Gustafson wrote:
> +1 Thanks for the KIP!
> 
> On Thu, Sep 3, 2020 at 12:25 PM Gokul Srinivas  wrote:
> 
>> Hi,
>>
>> I would like to call a vote on the following KIP.
>>
>> *KIP *-
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception
>>>
>>
>> TL;DR: This KIP proposes to throw a new, non-fatal exception whilst
>> aborting transactions with non-flushed data. This will help users
>> distinguish non-fatal errors and potentially retry the batch.
>>
>> Thanks,
>> -Gokul
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #49

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10432; LeaderEpochCache is incorrectly recovered for leader 
epoch 0 (#9219)


--
[...truncated 6.57 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

Re: [VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

[Guozhang Wang]

+1. Thanks!

Guozhang

On Tue, Sep 8, 2020 at 3:04 PM Ron Dagostino  wrote:

> +1 (non-binding) -- Thanks, Jason!
>
> Ron
>
> On Tue, Sep 8, 2020 at 2:04 PM Jason Gustafson  wrote:
>
> > Hi All,
> >
> > I'd like to start a vote on KIP-664:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions
> > .
> > Thanks for all the feedback!
> >
> > Best,
> > Jason
> >
>


-- 
-- Guozhang

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #49

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10432; LeaderEpochCache is incorrectly recovered for leader 
epoch 0 (#9219)


--
[...truncated 6.57 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Leah Thomas]

Hi Bill,

Could you also add KIP-450 to the release plan? It's been merged.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL

Cheers,
Leah

On Tue, Sep 8, 2020 at 9:32 AM Bill Bejeck  wrote:

> Hi Bruno,
>
> Thanks for letting me know, I've added KIP-662 to the release plan.
>
> -Bill
>
> On Mon, Sep 7, 2020 at 11:33 AM Bruno Cadonna  wrote:
>
> > Hi Bill,
> >
> > Could you add KIP-662 [1] to the release plan. The KIP has been already
> > implemented.
> >
> > Best,
> > Bruno
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-662%3A+Throw+Exception+when+Source+Topics+of+a+Streams+App+are+Deleted
> >
> > On 26.08.20 16:54, Bill Bejeck wrote:
> > > Greetings All!
> > >
> > > I've published a release plan at
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158872629
> > .
> > > I have included all of the KIPs that are currently approved, but I'm
> > happy
> > > to make any adjustments as necessary.
> > >
> > > The KIP freeze is on September 30 with a target release date of
> November
> > 6.
> > >
> > > Let me know if there are any objections.
> > >
> > > Thanks,
> > > Bill Bejeck
> > >
> > > On Fri, Aug 14, 2020 at 4:01 PM John Roesler 
> > wrote:
> > >
> > >> Thanks, Bill!
> > >> -John
> > >>
> > >> On Thu, 2020-08-13 at 15:19 -0700, Ismael Juma wrote:
> > >>> Thanks for volunteering Bill. :)
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Thu, Aug 13, 2020 at 3:13 PM Bill Bejeck 
> > wrote:
> > >>>
> >  Hi All,
> > 
> >  I'd like to volunteer to be the release manager for our next feature
> >  release, 2.7. If there are no objections, I'll send out the release
> > >> plan
> >  soon.
> > 
> >  Thanks,
> >  Bill Bejeck
> > 
> > >>
> > >>
> > >
> >
>

[jira] [Resolved] (KAFKA-5636) Add Sliding-Window support for Aggregations

[Leah Thomas (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leah Thomas resolved KAFKA-5636.

Resolution: Fixed

> Add Sliding-Window support for Aggregations
> ---
>
> Key: KAFKA-5636
> URL: https://issues.apache.org/jira/browse/KAFKA-5636
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Michael G. Noll
>Assignee: Leah Thomas
>Priority: Major
>  Labels: needs-kip
> Fix For: 2.7.0
>
>
> We support three windowing types for aggregations in the DSL right now:
>  * Tumbling windows
>  * Hopping windows (note: some stream processing tools call these "sliding 
> windows")
>  * Session windows
> Some users have expressed the need for sliding windows. While we do use 
> sliding windows for joins, we do not yet support sliding window aggregations 
> in the DSL



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #48

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10432; LeaderEpochCache is incorrectly recovered for leader 
epoch 0 (#9219)


--
[...truncated 3.26 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 

Re: [VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

[Ron Dagostino]

+1 (non-binding) -- Thanks, Jason!

Ron

On Tue, Sep 8, 2020 at 2:04 PM Jason Gustafson  wrote:

> Hi All,
>
> I'd like to start a vote on KIP-664:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions
> .
> Thanks for all the feedback!
>
> Best,
> Jason
>

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

[Jason Gustafson]

+1 Thanks for the KIP!

On Thu, Sep 3, 2020 at 12:25 PM Gokul Srinivas  wrote:

> Hi,
>
> I would like to call a vote on the following KIP.
>
> *KIP *-
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception
> >
>
> TL;DR: This KIP proposes to throw a new, non-fatal exception whilst
> aborting transactions with non-flushed data. This will help users
> distinguish non-fatal errors and potentially retry the batch.
>
> Thanks,
> -Gokul
>
>

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Sophie Blee-Goldman]

Ah, I forgot about localThreadsMetadata(). In that. case I agree, there's
no reason
to introduce a new method when we can get both the names and number of all
running threads from this.

I assume that we would update localThreadsMetadata to only return currently
alive threads as part of this KIP -- at a quick glance, it seems like we
don't do
any pruning of dead threads at the moment

On Tue, Sep 8, 2020 at 1:58 PM Matthias J. Sax  wrote:

> I am not sure if we need a new method? There is already
> `localThreadsMetadata()`. What do we gain by adding a new one?
>
> Returning the thread's name (as `Optional`) for both add() and
> remove() is fine with me.
>
>
> -Matthias
>
> On 9/8/20 12:58 PM, Sophie Blee-Goldman wrote:
> > Sorry Bruno, I think I missed the end of your message with the
> > numberOfAliveStreamThreads()
> > proposal. I agree, that would be better than the alternatives I listed.
> > That said:
> >
> >> They rather suggest that the method returns a list of handles to the
> > stream threads.
> >
> > I hadn't thought of that originally, but now that you mention it, this
> > might be a good idea.
> > I don't think we should return actual handles on the threads, but maybe a
> > list of the thread
> > names rather than a single number of currently alive threads.
> >
> > Since we seem to think it would be difficult if not impossible to keep
> > track of the number
> > of running stream threads, we should apply the same reasoning to the
> names
> > and not
> > assume the user can/will keep track of every thread returned by
> > addStreamThread() or
> > removeStreamThread(). Users should generally take any required action
> > immediately
> > after adding/removing the thread -- eg deregistering the thread metrics
> --
> > but it might
> > still be useful to provide a convenience method listing all of the
> current
> > threads
> >
> > And of course you could still get the number of threads easily by
> invoking
> > size() on the
> > returned list (or ordered set?).
> >
> > On Tue, Sep 8, 2020 at 12:16 PM Bruno Cadonna 
> wrote:
> >
> >> Thank you again for the feedback Sophie!
> >>
> >> As I tried to point out in my previous e-mail, removing a stream thread
> >> from a Kafka Streams client that does not have alive stream threads is
> >> nothing exceptional for the client per se. However, it can become
> >> exceptional within the context of the user. For example, if users want
> >> to remove a stream thread from a client without alive stream threads
> >> because one if their metrics say so, then this is exceptional in the
> >> context of that user metric not in the context of the Kafka Streams
> >> client. In that case, users should throw an exception and handle it.
> >>
> >> Regarding returning null, I do not like to return null because from a
> >> development point of view there is no distinction between returning null
> >> because we have a bug in the code or returning null because there are no
> >> alive stream threads. Additionally, Optional makes it more
> >> explicit that the result could also be empty.
> >>
> >> Thank you for the alternative method names! However, with the names you
> >> propose it is not immediately clear that the method returns an amount of
> >> stream threads. They rather suggest that the method returns a list of
> >> handles to the stream threads. I chose to use "aliveStreamThreads" to be
> >> consistent with the client-level metric "alive-stream-threads" which
> >> reports the same number of stream threads that
> >> numberOfAliveStreamThreads() should report. If others also think that
> >> the proposed name in the KIP is too clumsy, I am open to rename it,
> though.
> >>
> >> Best,
> >> Bruno
> >>
> >>
> >> On 08.09.20 20:12, Sophie Blee-Goldman wrote:
>  it's never a good sign when the discussion moves into the vote thread
> >>>
> >>> Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads
> >> strikes
> >>> again.
> >>> Thanks for redirecting me Bruno
> >>>
> >>> I suppose it's unfair to expect the callers to keep perfect track of
> the
> >>> current
> >>>   number of stream threads, but it also seems like you shouldn't be
> >> calling
> >>> removeStreamThread() when there are no threads left. Either you're just
> >>> haphazardly removing threads and could unintentionally slip into a
> state
> >> of
> >>> no
> >>> running threads without realizing it, or more realistically, you're
> >>> carefully
> >>> removing threads based on some metric(s) that convey whether the system
> >> is
> >>> over or under-provisioned. If your metrics say you're over-provisioned
> >> but
> >>> there's
> >>> not one thread running, well, that certainly sounds exceptional to me.
> Or
> >>> you might
> >>> be right in that the cluster is over-provisioned but have just been
> >>> directing the
> >>> removeStreamThread() and addStreamThread() calls to instances at
> random,
> >> and
> >>> end up with one massive instance and one with no threads at all. Again,
> >>> this
> >>> probably 

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Matthias J. Sax]

From my understanding, the KIP aims for the case when a user does not
control the code, eg, when using the command line consumer (or similar
tools).

If the user writes code, we should always encourage her to instantiate
the deserializer explicitly and not relying on reflection+config.

I also don't think that the `default` prefix does make sense, as it
indicates that there might be a non-default. However, IMHO, we should
not allow "overwrite semantics" but rather throw an exception if the
config is set and a window size is provided via the constructor. We
should not allow to mix-and-match both and should stick to a strict
either-or pattern.


-Matthias

On 9/8/20 11:52 AM, Guozhang Wang wrote:
> Hi Sophie,
> 
> Seems I do have some mis-understanding of the KIP's motivation here :) Just
> for clarification my reasoning is that:
> 
> 1) today Streams itself never uses a windowed deserializer itself since its
> built-in operators only need the serializer and users do not need to
> override it, plus standby / restore active tasks would just copy the bytes
> directly. So this KIP's motivation is not for Stream's own code anyways.
> 
> 2) It is only when user specified serde is missing the window size, which
> is either when a) one is trying to read a source topic as windowed records
> in Streams, this is a big blocker for KIP-300, and when b) one is trying to
> read a topic as windowed records in Consumer, we would have issues if users
> fail to use the appropriate serde constructs.
> 
> I thought the main motivation of this KIP is for 2.a), in which we would
> just encourage the users to use the right constructor with the window size
> by deprecating the other constructs. But I'm not sure how this would help
> with 2.b) since the proposal is on adding to StreamsConfig. If it is the
> case, then I agree that probably we can just not add an extra config but
> just deprecating the constructs.
> 
> 
> Guozhang
> 
> 
> 
> 
> 
> On Tue, Sep 8, 2020 at 10:50 AM Sophie Blee-Goldman 
> wrote:
> 
>> Hey Guozhang & Leah,
>>
>> I want to push back a bit on the assumption that we would fall back on this
>> config
>> in the case of an unspecified window size in a Streams serde. I don't think
>> it should
>> be a default at all, either in name or in effect. To borrow the rhetorical
>> question that
>> John raised earlier: what is the default window size of an application?
>>
>> Personally, I agree that that doesn't make much sense. Sure, if you only
>> have a single
>> windowed operation in your app then you could just specify the window size
>> by config,
>> but how is that any more ergonomic than specifying the window size in the
>> Serde's
>> constructor? If anything, it seems worse to put physical and mental
>> distance between
>> the specification and the actual usage of such parameters. What if you add
>> another
>> windowed operation later, with a different size, and forget to specify the
>> new size in
>> the new Serde? Or what if you never specify a default window size config at
>> all and
>> accidentally end up using the default config value of Long.MAX_VALUE?
>> Avoiding this
>> possibility was/is one of the main motivations of this KIP, and the whole
>> point of
>> deprecating the TimeWindowedSerde(innerClass) constructor.
>>
>> I actually would have advocated to remove this config entirely, but as John
>> pointed
>> out, we still need it to configure things like the console consumer
>>
>> On Tue, Sep 8, 2020 at 10:40 AM Leah Thomas  wrote:
>>
>>> Hi Guozhang,
>>>
>>> Yes, the config would read them as a single window size. I think this
>>> relates to John's comments about having variably sized windows, which
>> this
>>> config doesn't handle. I like the name change and updated the wiki to
>>> reflect that, and to clarify that the default value will still be
>>> Long.MAX_VALUE.
>>>
>>> Thanks for your feedback!
>>> Leah
>>>
>>> On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang 
>> wrote:
>>>
 Hello Leah,

 Thanks for initiating this. I just have one minor clarification
>> question
 here: the config "window.size.ms" seems to be used as the default
>> window
 size when reading from a topic that represents windowed records right?
>>> I.e.
 if there are multiple topics that represent windowed records but their
 window sizes are different, with this config we can only read them
>> with a
 single window size? If yes, could we rename the config as "
 default.window.size.ms" and make that clear in the description as
>> well?
 Also we'd better also include its default value which I think would
>> still
 be MAX_VALUE for compatibility.


 Guozhang


 On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas 
>> wrote:

> Hey all,
>
> We should be good to wrap up voting now that the discussion has been
> resolved.
>
> Cheers,
> Leah
>
> On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax 
>>> wrote:
>
>> +1 (binding)
>>
>> 

[jira] [Created] (KAFKA-10471) TimeIndex handling may cause data loss in certain back to back failure

[Rohit Shekhar (Jira)]

Rohit Shekhar created KAFKA-10471:
-

 Summary: TimeIndex handling may cause data loss in certain back to 
back failure
 Key: KAFKA-10471
 URL: https://issues.apache.org/jira/browse/KAFKA-10471
 Project: Kafka
  Issue Type: Bug
  Components: core, log
Reporter: Rohit Shekhar


# Active segment for log A going clean shutdown - trim the time index to the 
latest fill value, set the clean shutdown marker.
 # Broker restarts, loading logs - no recovery due to clean shutdown marker, 
log A recovers with the previous active segment as current. It also resized the 
TimeIndex to the max.
 #  Before all the log loads, the broker had a hard shutdown causing a clean 
shutdown marker left as is.
 #  Broker restarts, log A skips recovery due to the presence of a clean 
shutdown marker but the TimeIndex file assumes the resized file from the 
previous instance is all full (it assumes either file is newly created or is 
full with valid value).
 # The first append to the active segment will result in roll and TimeIndex 
will be rolled with the timestamp value of the last valid entry (0)
 # Segment.the largest timestamp gives 0 (this can cause premature deletion of 
data due to retention.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Matthias J. Sax]

I am not sure if we need a new method? There is already
`localThreadsMetadata()`. What do we gain by adding a new one?

Returning the thread's name (as `Optional`) for both add() and
remove() is fine with me.


-Matthias

On 9/8/20 12:58 PM, Sophie Blee-Goldman wrote:
> Sorry Bruno, I think I missed the end of your message with the
> numberOfAliveStreamThreads()
> proposal. I agree, that would be better than the alternatives I listed.
> That said:
> 
>> They rather suggest that the method returns a list of handles to the
> stream threads.
> 
> I hadn't thought of that originally, but now that you mention it, this
> might be a good idea.
> I don't think we should return actual handles on the threads, but maybe a
> list of the thread
> names rather than a single number of currently alive threads.
> 
> Since we seem to think it would be difficult if not impossible to keep
> track of the number
> of running stream threads, we should apply the same reasoning to the names
> and not
> assume the user can/will keep track of every thread returned by
> addStreamThread() or
> removeStreamThread(). Users should generally take any required action
> immediately
> after adding/removing the thread -- eg deregistering the thread metrics --
> but it might
> still be useful to provide a convenience method listing all of the current
> threads
> 
> And of course you could still get the number of threads easily by invoking
> size() on the
> returned list (or ordered set?).
> 
> On Tue, Sep 8, 2020 at 12:16 PM Bruno Cadonna  wrote:
> 
>> Thank you again for the feedback Sophie!
>>
>> As I tried to point out in my previous e-mail, removing a stream thread
>> from a Kafka Streams client that does not have alive stream threads is
>> nothing exceptional for the client per se. However, it can become
>> exceptional within the context of the user. For example, if users want
>> to remove a stream thread from a client without alive stream threads
>> because one if their metrics say so, then this is exceptional in the
>> context of that user metric not in the context of the Kafka Streams
>> client. In that case, users should throw an exception and handle it.
>>
>> Regarding returning null, I do not like to return null because from a
>> development point of view there is no distinction between returning null
>> because we have a bug in the code or returning null because there are no
>> alive stream threads. Additionally, Optional makes it more
>> explicit that the result could also be empty.
>>
>> Thank you for the alternative method names! However, with the names you
>> propose it is not immediately clear that the method returns an amount of
>> stream threads. They rather suggest that the method returns a list of
>> handles to the stream threads. I chose to use "aliveStreamThreads" to be
>> consistent with the client-level metric "alive-stream-threads" which
>> reports the same number of stream threads that
>> numberOfAliveStreamThreads() should report. If others also think that
>> the proposed name in the KIP is too clumsy, I am open to rename it, though.
>>
>> Best,
>> Bruno
>>
>>
>> On 08.09.20 20:12, Sophie Blee-Goldman wrote:
 it's never a good sign when the discussion moves into the vote thread
>>>
>>> Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads
>> strikes
>>> again.
>>> Thanks for redirecting me Bruno
>>>
>>> I suppose it's unfair to expect the callers to keep perfect track of the
>>> current
>>>   number of stream threads, but it also seems like you shouldn't be
>> calling
>>> removeStreamThread() when there are no threads left. Either you're just
>>> haphazardly removing threads and could unintentionally slip into a state
>> of
>>> no
>>> running threads without realizing it, or more realistically, you're
>>> carefully
>>> removing threads based on some metric(s) that convey whether the system
>> is
>>> over or under-provisioned. If your metrics say you're over-provisioned
>> but
>>> there's
>>> not one thread running, well, that certainly sounds exceptional to me. Or
>>> you might
>>> be right in that the cluster is over-provisioned but have just been
>>> directing the
>>> removeStreamThread() and addStreamThread() calls to instances at random,
>> and
>>> end up with one massive instance and one with no threads at all. Again,
>>> this
>>> probably merits some human intervention (or system redesign)
>>>
>>> That said, I don't think there's any real harm to just returning null in
>>> this case, but I hope
>>> that users would pay attention to this since it seems likely to indicate
>>> something has gone
>>> seriously wrong. I suppose Optional would be a reasonable
>>> compromise.
>>>
>>> As for the method name, what about activeStreamThreads() or
>>> liveStreamThreads() ?
>>>
>>> On Mon, Sep 7, 2020 at 1:45 AM Bruno Cadonna  wrote:
>>>
 Hi John,

 I agree with you except for checking null. I would rather prefer to use
 Optional as the return type to both methods.

 I changed the 

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Sophie Blee-Goldman]

Sorry Bruno, I think I missed the end of your message with the
numberOfAliveStreamThreads()
proposal. I agree, that would be better than the alternatives I listed.
That said:

> They rather suggest that the method returns a list of handles to the
stream threads.

I hadn't thought of that originally, but now that you mention it, this
might be a good idea.
I don't think we should return actual handles on the threads, but maybe a
list of the thread
names rather than a single number of currently alive threads.

Since we seem to think it would be difficult if not impossible to keep
track of the number
of running stream threads, we should apply the same reasoning to the names
and not
assume the user can/will keep track of every thread returned by
addStreamThread() or
removeStreamThread(). Users should generally take any required action
immediately
after adding/removing the thread -- eg deregistering the thread metrics --
but it might
still be useful to provide a convenience method listing all of the current
threads

And of course you could still get the number of threads easily by invoking
size() on the
returned list (or ordered set?).

On Tue, Sep 8, 2020 at 12:16 PM Bruno Cadonna  wrote:

> Thank you again for the feedback Sophie!
>
> As I tried to point out in my previous e-mail, removing a stream thread
> from a Kafka Streams client that does not have alive stream threads is
> nothing exceptional for the client per se. However, it can become
> exceptional within the context of the user. For example, if users want
> to remove a stream thread from a client without alive stream threads
> because one if their metrics say so, then this is exceptional in the
> context of that user metric not in the context of the Kafka Streams
> client. In that case, users should throw an exception and handle it.
>
> Regarding returning null, I do not like to return null because from a
> development point of view there is no distinction between returning null
> because we have a bug in the code or returning null because there are no
> alive stream threads. Additionally, Optional makes it more
> explicit that the result could also be empty.
>
> Thank you for the alternative method names! However, with the names you
> propose it is not immediately clear that the method returns an amount of
> stream threads. They rather suggest that the method returns a list of
> handles to the stream threads. I chose to use "aliveStreamThreads" to be
> consistent with the client-level metric "alive-stream-threads" which
> reports the same number of stream threads that
> numberOfAliveStreamThreads() should report. If others also think that
> the proposed name in the KIP is too clumsy, I am open to rename it, though.
>
> Best,
> Bruno
>
>
> On 08.09.20 20:12, Sophie Blee-Goldman wrote:
> >> it's never a good sign when the discussion moves into the vote thread
> >
> > Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads
> strikes
> > again.
> > Thanks for redirecting me Bruno
> >
> > I suppose it's unfair to expect the callers to keep perfect track of the
> > current
> >   number of stream threads, but it also seems like you shouldn't be
> calling
> > removeStreamThread() when there are no threads left. Either you're just
> > haphazardly removing threads and could unintentionally slip into a state
> of
> > no
> > running threads without realizing it, or more realistically, you're
> > carefully
> > removing threads based on some metric(s) that convey whether the system
> is
> > over or under-provisioned. If your metrics say you're over-provisioned
> but
> > there's
> > not one thread running, well, that certainly sounds exceptional to me. Or
> > you might
> > be right in that the cluster is over-provisioned but have just been
> > directing the
> > removeStreamThread() and addStreamThread() calls to instances at random,
> and
> > end up with one massive instance and one with no threads at all. Again,
> > this
> > probably merits some human intervention (or system redesign)
> >
> > That said, I don't think there's any real harm to just returning null in
> > this case, but I hope
> > that users would pay attention to this since it seems likely to indicate
> > something has gone
> > seriously wrong. I suppose Optional would be a reasonable
> > compromise.
> >
> > As for the method name, what about activeStreamThreads() or
> > liveStreamThreads() ?
> >
> > On Mon, Sep 7, 2020 at 1:45 AM Bruno Cadonna  wrote:
> >
> >> Hi John,
> >>
> >> I agree with you except for checking null. I would rather prefer to use
> >> Optional as the return type to both methods.
> >>
> >> I changed the subject from [VOTE] to [DISCUSS] so that we can follow up
> >> in the discussion thread.
> >>
> >> Best,
> >> Bruno
> >>
> >> On 04.09.20 23:12, John Roesler wrote:
> >>> Hi Sophie,
> >>>
> >>> Uh, oh, it's never a good sign when the discussion moves
> >>> into the vote thread :)
> >>>
> >>> I agree with you, it seems like a good touch for
> >>> removeStreamThread() 

[jira] [Resolved] (KAFKA-10432) LeaderEpochCache is incorrectly recovered on segment recovery for epoch 0

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-10432.
-
Fix Version/s: 2.6.1
   Resolution: Fixed

> LeaderEpochCache is incorrectly recovered on segment recovery for epoch 0
> -
>
> Key: KAFKA-10432
> URL: https://issues.apache.org/jira/browse/KAFKA-10432
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0, 2.5.0, 2.6.0
>Reporter: Lucas Bradstreet
>Priority: Major
> Fix For: 2.6.1
>
>
> I added some functionality to the system tests to compare epoch cache 
> lineages ([https://github.com/apache/kafka/pull/9213]), and I found a bug in 
> leader epoch cache recovery.
> The test hard kills a broker and the cache hasn't been flushed yet, and then 
> it starts up and goes through log recovery. After recovery there is 
> divergence in the epoch caches for epoch 0:
> {noformat}
> AssertionError: leader epochs for output-topic-1 didn't match
>  [{0: 9393L, 2: 9441L, 4: 42656L},
>  {0: 0L, 2: 9441L, 4: 42656L}, 
>  {0: 0L, 2: 9441L, 4: 42656L}]
>   
>   
> {noformat}
> The cache is supposed to include the offset for epoch 0 but in recovery it 
> skips it 
> [https://github.com/apache/kafka/blob/487b3682ebe0eefde3445b37ee72956451a9d15e/core/src/main/scala/kafka/log/LogSegment.scala#L364]
>  due to 
> [https://github.com/apache/kafka/commit/d152989f26f51b9004b881397db818ad6eaf0392].
>  Then it stamps the epoch with a later offset when fetching from the leader.
> I'm not sure why the recovery code includes the condition 
> `batch.partitionLeaderEpoch > 0`. I discussed this with Jason Gustafson and 
> he believes it may have been intended to avoid assigning negative epochs but 
> is not sure why it was added. None of the tests fail with this check removed.
> {noformat}
>   leaderEpochCache.foreach { cache =>
> if (batch.partitionLeaderEpoch > 0 && 
> cache.latestEpoch.forall(batch.partitionLeaderEpoch > _))
>   cache.assign(batch.partitionLeaderEpoch, batch.baseOffset)
>   }
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [kafka-site] ableegoldman commented on a change in pull request #301: Redo 2.6 docs commit

[GitBox]

ableegoldman commented on a change in pull request #301:
URL: https://github.com/apache/kafka-site/pull/301#discussion_r485154723



##
File path: 26/streams/developer-guide/memory-mgmt.html
##
@@ -202,7 +202,10 @@ RocksDB}
 }
   
-1. INDEX_FILTER_BLOCK_RATIO can be used to set a 
fraction of the block cache to set aside for "high priority" (aka index and 
filter) blocks, preventing them from being evicted by data blocks. See the full 
signature of the https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/LRUCache.java#L72;>LRUCache
 constructor. 
+1. INDEX_FILTER_BLOCK_RATIO can be used to set a 
fraction of the block cache to set aside for "high priority" (aka index and 
filter) blocks, preventing them from being evicted by data blocks. See the full 
signature of the https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/LRUCache.java#L72;>LRUCache
 constructor.
+  NOTE: the boolean parameter in the cache constructor lets you 
control whether the cache should enforce a strict memory limit by failing the 
read or iteration in the rare cases where it might go larger than its capacity. 
Due to a
+  https://github.com/facebook/rocksdb/issues/6247;>bug in RocksDB, this 
option cannot be used
+  if the write buffer memory is also counted against the cache. If you 
set this to true, you should NOT pass the cache in to the 
WriteBufferManager and just control the write buffer and cache 
memory separately.

Review comment:
   Nope, this is correct. My original commit had a few other fixes on the 
side in addition to the KIP-441/KIP-613 docs
   
   As far as I know this has always been the case in every RocksDB version 
we've depended on but didn't discover the bug until just recently. (I believe 
it actually is fixed in some newer versions, so we may be able to remove this 
again once we upgrade)
   
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] ableegoldman commented on a change in pull request #301: Redo 2.6 docs commit

[GitBox]

ableegoldman commented on a change in pull request #301:
URL: https://github.com/apache/kafka-site/pull/301#discussion_r485154723



##
File path: 26/streams/developer-guide/memory-mgmt.html
##
@@ -202,7 +202,10 @@ RocksDB}
 }
   
-1. INDEX_FILTER_BLOCK_RATIO can be used to set a 
fraction of the block cache to set aside for "high priority" (aka index and 
filter) blocks, preventing them from being evicted by data blocks. See the full 
signature of the https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/LRUCache.java#L72;>LRUCache
 constructor. 
+1. INDEX_FILTER_BLOCK_RATIO can be used to set a 
fraction of the block cache to set aside for "high priority" (aka index and 
filter) blocks, preventing them from being evicted by data blocks. See the full 
signature of the https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/LRUCache.java#L72;>LRUCache
 constructor.
+  NOTE: the boolean parameter in the cache constructor lets you 
control whether the cache should enforce a strict memory limit by failing the 
read or iteration in the rare cases where it might go larger than its capacity. 
Due to a
+  https://github.com/facebook/rocksdb/issues/6247;>bug in RocksDB, this 
option cannot be used
+  if the write buffer memory is also counted against the cache. If you 
set this to true, you should NOT pass the cache in to the 
WriteBufferManager and just control the write buffer and cache 
memory separately.

Review comment:
   Nope, this is correct. As far as I know this has always been the case in 
every RocksDB version we've depended on but didn't discover the bug until just 
recently. (I believe it actually is fixed in some newer versions, so we may be 
able to remove this again once we upgrade)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] vvcephei commented on a change in pull request #301: Redo 2.6 docs commit

[GitBox]

vvcephei commented on a change in pull request #301:
URL: https://github.com/apache/kafka-site/pull/301#discussion_r485142442



##
File path: 26/streams/developer-guide/memory-mgmt.html
##
@@ -202,7 +202,10 @@ RocksDB}
 }
   
-1. INDEX_FILTER_BLOCK_RATIO can be used to set a 
fraction of the block cache to set aside for "high priority" (aka index and 
filter) blocks, preventing them from being evicted by data blocks. See the full 
signature of the https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/LRUCache.java#L72;>LRUCache
 constructor. 
+1. INDEX_FILTER_BLOCK_RATIO can be used to set a 
fraction of the block cache to set aside for "high priority" (aka index and 
filter) blocks, preventing them from being evicted by data blocks. See the full 
signature of the https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/LRUCache.java#L72;>LRUCache
 constructor.
+  NOTE: the boolean parameter in the cache constructor lets you 
control whether the cache should enforce a strict memory limit by failing the 
read or iteration in the rare cases where it might go larger than its capacity. 
Due to a
+  https://github.com/facebook/rocksdb/issues/6247;>bug in RocksDB, this 
option cannot be used
+  if the write buffer memory is also counted against the cache. If you 
set this to true, you should NOT pass the cache in to the 
WriteBufferManager and just control the write buffer and cache 
memory separately.

Review comment:
   It looks like this (and some other stuff in the config-streams file) has 
come along for the ride. We should make sure it's referencing behavior that was 
released, not behavior that will be released next. Can you double-check that 
this actually is in reference to 2.6.0, and not 2.6.1+?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Bruno Cadonna]

Thank you again for the feedback Sophie!

As I tried to point out in my previous e-mail, removing a stream thread 
from a Kafka Streams client that does not have alive stream threads is 
nothing exceptional for the client per se. However, it can become 
exceptional within the context of the user. For example, if users want 
to remove a stream thread from a client without alive stream threads 
because one if their metrics say so, then this is exceptional in the 
context of that user metric not in the context of the Kafka Streams 
client. In that case, users should throw an exception and handle it.


Regarding returning null, I do not like to return null because from a 
development point of view there is no distinction between returning null 
because we have a bug in the code or returning null because there are no 
alive stream threads. Additionally, Optional makes it more 
explicit that the result could also be empty.


Thank you for the alternative method names! However, with the names you 
propose it is not immediately clear that the method returns an amount of 
stream threads. They rather suggest that the method returns a list of 
handles to the stream threads. I chose to use "aliveStreamThreads" to be 
consistent with the client-level metric "alive-stream-threads" which 
reports the same number of stream threads that 
numberOfAliveStreamThreads() should report. If others also think that 
the proposed name in the KIP is too clumsy, I am open to rename it, though.


Best,
Bruno


On 08.09.20 20:12, Sophie Blee-Goldman wrote:

it's never a good sign when the discussion moves into the vote thread


Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads strikes
again.
Thanks for redirecting me Bruno

I suppose it's unfair to expect the callers to keep perfect track of the
current
  number of stream threads, but it also seems like you shouldn't be calling
removeStreamThread() when there are no threads left. Either you're just
haphazardly removing threads and could unintentionally slip into a state of
no
running threads without realizing it, or more realistically, you're
carefully
removing threads based on some metric(s) that convey whether the system is
over or under-provisioned. If your metrics say you're over-provisioned but
there's
not one thread running, well, that certainly sounds exceptional to me. Or
you might
be right in that the cluster is over-provisioned but have just been
directing the
removeStreamThread() and addStreamThread() calls to instances at random, and
end up with one massive instance and one with no threads at all. Again,
this
probably merits some human intervention (or system redesign)

That said, I don't think there's any real harm to just returning null in
this case, but I hope
that users would pay attention to this since it seems likely to indicate
something has gone
seriously wrong. I suppose Optional would be a reasonable
compromise.

As for the method name, what about activeStreamThreads() or
liveStreamThreads() ?

On Mon, Sep 7, 2020 at 1:45 AM Bruno Cadonna  wrote:


Hi John,

I agree with you except for checking null. I would rather prefer to use
Optional as the return type to both methods.

I changed the subject from [VOTE] to [DISCUSS] so that we can follow up
in the discussion thread.

Best,
Bruno

On 04.09.20 23:12, John Roesler wrote:

Hi Sophie,

Uh, oh, it's never a good sign when the discussion moves
into the vote thread :)

I agree with you, it seems like a good touch for
removeStreamThread() to return the name of the thread that
got removed, rather than a boolean flag. Maybe the return
value would be `null` if there is no thread to remove.

If we go that way, I'd suggest that addStreamThread() also
return the name of the newly created thread, or null if no
thread can be created right now.

I'm not completely sure if I think that callers of this
method would know exactly how many threads there are. Sure,
if a human being is sitting there looking at the metrics or
logs and decides to call the method, it would work out, but
I'd expect this kind of method to find its way into
automated tooling that reacts to things like current system
load or resource saturation. Those kinds of toolchains often
are part of a distributed system, and it's probably not that
easy to guarantee that the thread count they observe is
fully consistent with the number of threads that are
actually running. Therefore, an in-situ `int
numStreamThreads()` method might not be a bad idea. Then
again, it seems sort of optional. A caller can catch an
exception or react to a `null` return value just the same
either way. Having both add/remove methods behave similarly
is probably more valuable.

Thanks,
-John


On Thu, 2020-09-03 at 12:15 -0700, Sophie Blee-Goldman
wrote:

Hey, sorry for the late reply, I just have one minor suggestion. Since

we

don't
make any guarantees about which thread gets removed or allow the user to
specify, I think we should return either the index or full name of 

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Guozhang Wang]

Hi Sophie,

Seems I do have some mis-understanding of the KIP's motivation here :) Just
for clarification my reasoning is that:

1) today Streams itself never uses a windowed deserializer itself since its
built-in operators only need the serializer and users do not need to
override it, plus standby / restore active tasks would just copy the bytes
directly. So this KIP's motivation is not for Stream's own code anyways.

2) It is only when user specified serde is missing the window size, which
is either when a) one is trying to read a source topic as windowed records
in Streams, this is a big blocker for KIP-300, and when b) one is trying to
read a topic as windowed records in Consumer, we would have issues if users
fail to use the appropriate serde constructs.

I thought the main motivation of this KIP is for 2.a), in which we would
just encourage the users to use the right constructor with the window size
by deprecating the other constructs. But I'm not sure how this would help
with 2.b) since the proposal is on adding to StreamsConfig. If it is the
case, then I agree that probably we can just not add an extra config but
just deprecating the constructs.


Guozhang





On Tue, Sep 8, 2020 at 10:50 AM Sophie Blee-Goldman 
wrote:

> Hey Guozhang & Leah,
>
> I want to push back a bit on the assumption that we would fall back on this
> config
> in the case of an unspecified window size in a Streams serde. I don't think
> it should
> be a default at all, either in name or in effect. To borrow the rhetorical
> question that
> John raised earlier: what is the default window size of an application?
>
> Personally, I agree that that doesn't make much sense. Sure, if you only
> have a single
> windowed operation in your app then you could just specify the window size
> by config,
> but how is that any more ergonomic than specifying the window size in the
> Serde's
> constructor? If anything, it seems worse to put physical and mental
> distance between
> the specification and the actual usage of such parameters. What if you add
> another
> windowed operation later, with a different size, and forget to specify the
> new size in
> the new Serde? Or what if you never specify a default window size config at
> all and
> accidentally end up using the default config value of Long.MAX_VALUE?
> Avoiding this
> possibility was/is one of the main motivations of this KIP, and the whole
> point of
> deprecating the TimeWindowedSerde(innerClass) constructor.
>
> I actually would have advocated to remove this config entirely, but as John
> pointed
> out, we still need it to configure things like the console consumer
>
> On Tue, Sep 8, 2020 at 10:40 AM Leah Thomas  wrote:
>
> > Hi Guozhang,
> >
> > Yes, the config would read them as a single window size. I think this
> > relates to John's comments about having variably sized windows, which
> this
> > config doesn't handle. I like the name change and updated the wiki to
> > reflect that, and to clarify that the default value will still be
> > Long.MAX_VALUE.
> >
> > Thanks for your feedback!
> > Leah
> >
> > On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang 
> wrote:
> >
> > > Hello Leah,
> > >
> > > Thanks for initiating this. I just have one minor clarification
> question
> > > here: the config "window.size.ms" seems to be used as the default
> window
> > > size when reading from a topic that represents windowed records right?
> > I.e.
> > > if there are multiple topics that represent windowed records but their
> > > window sizes are different, with this config we can only read them
> with a
> > > single window size? If yes, could we rename the config as "
> > > default.window.size.ms" and make that clear in the description as
> well?
> > > Also we'd better also include its default value which I think would
> still
> > > be MAX_VALUE for compatibility.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas 
> wrote:
> > >
> > > > Hey all,
> > > >
> > > > We should be good to wrap up voting now that the discussion has been
> > > > resolved.
> > > >
> > > > Cheers,
> > > > Leah
> > > >
> > > > On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax 
> > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On 8/26/20 8:02 AM, John Roesler wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I've just sent a new message to the DISCUSS thread. We
> > > > > > forgot to include the Scala API in the proposal.
> > > > > >
> > > > > > Thanks,
> > > > > > -John
> > > > > >
> > > > > > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > > > > > wrote:
> > > > > >> Thanks for the KIP! +1 (non-binding)
> > > > > >>
> > > > > >> Sophie
> > > > > >>
> > > > > >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler <
> vvcep...@apache.org
> > >
> > > > > wrote:
> > > > > >>
> > > > > >>> Thanks Leah,
> > > > > >>> I’m +1 (binding)
> > > > > >>>
> > > > > >>> -John
> > > > > >>>
> > > > > >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
> > > > >  Hi everyone,
> > > 

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Leah Thomas]

Hey Sophie,

Are you advocating that we change the name back to *window.size.ms
*?

I wasn't thinking that by including the config we would steer people away
from using the constructors that pass in a window size, so in that sense I
suppose the config isn't "default." I think that by deprecating the
TimeWindowedSerde(innerClass) constructor, as you mentioned, we'll steer
clear from some of the issues you brought up. In my mind, the first tier of
usage is to set the window size directly, and then after that is to use the
config, and if neither of those happen, then we would fall back on
Long.MAX_VALUE. I suppose in this sense, "default" is an imprecise term,
but indicates that if the proper constructor/set up is not used, there will
be a "default" window size that the user still sets. It seems like the
console consumer has us somewhat stuck with a new config, but I agree that
the new constructors should be relied upon before the config is. On that
line, what about a name like *backup.window.size.ms
* instead of default?

Cheers,
Leah

On Tue, Sep 8, 2020 at 12:50 PM Sophie Blee-Goldman 
wrote:

> Hey Guozhang & Leah,
>
> I want to push back a bit on the assumption that we would fall back on this
> config
> in the case of an unspecified window size in a Streams serde. I don't think
> it should
> be a default at all, either in name or in effect. To borrow the rhetorical
> question that
> John raised earlier: what is the default window size of an application?
>
> Personally, I agree that that doesn't make much sense. Sure, if you only
> have a single
> windowed operation in your app then you could just specify the window size
> by config,
> but how is that any more ergonomic than specifying the window size in the
> Serde's
> constructor? If anything, it seems worse to put physical and mental
> distance between
> the specification and the actual usage of such parameters. What if you add
> another
> windowed operation later, with a different size, and forget to specify the
> new size in
> the new Serde? Or what if you never specify a default window size config at
> all and
> accidentally end up using the default config value of Long.MAX_VALUE?
> Avoiding this
> possibility was/is one of the main motivations of this KIP, and the whole
> point of
> deprecating the TimeWindowedSerde(innerClass) constructor.
>
> I actually would have advocated to remove this config entirely, but as John
> pointed
> out, we still need it to configure things like the console consumer
>
> On Tue, Sep 8, 2020 at 10:40 AM Leah Thomas  wrote:
>
> > Hi Guozhang,
> >
> > Yes, the config would read them as a single window size. I think this
> > relates to John's comments about having variably sized windows, which
> this
> > config doesn't handle. I like the name change and updated the wiki to
> > reflect that, and to clarify that the default value will still be
> > Long.MAX_VALUE.
> >
> > Thanks for your feedback!
> > Leah
> >
> > On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang 
> wrote:
> >
> > > Hello Leah,
> > >
> > > Thanks for initiating this. I just have one minor clarification
> question
> > > here: the config "window.size.ms" seems to be used as the default
> window
> > > size when reading from a topic that represents windowed records right?
> > I.e.
> > > if there are multiple topics that represent windowed records but their
> > > window sizes are different, with this config we can only read them
> with a
> > > single window size? If yes, could we rename the config as "
> > > default.window.size.ms" and make that clear in the description as
> well?
> > > Also we'd better also include its default value which I think would
> still
> > > be MAX_VALUE for compatibility.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas 
> wrote:
> > >
> > > > Hey all,
> > > >
> > > > We should be good to wrap up voting now that the discussion has been
> > > > resolved.
> > > >
> > > > Cheers,
> > > > Leah
> > > >
> > > > On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax 
> > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On 8/26/20 8:02 AM, John Roesler wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I've just sent a new message to the DISCUSS thread. We
> > > > > > forgot to include the Scala API in the proposal.
> > > > > >
> > > > > > Thanks,
> > > > > > -John
> > > > > >
> > > > > > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > > > > > wrote:
> > > > > >> Thanks for the KIP! +1 (non-binding)
> > > > > >>
> > > > > >> Sophie
> > > > > >>
> > > > > >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler <
> vvcep...@apache.org
> > >
> > > > > wrote:
> > > > > >>
> > > > > >>> Thanks Leah,
> > > > > >>> I’m +1 (binding)
> > > > > >>>
> > > > > >>> -John
> > > > > >>>
> > > > > >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
> > > > >  Hi everyone,
> > > > > 
> > > > >  I'd like to kick-off the vote for KIP-659: Improve
> > > > > 

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #48

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #47

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update scala default version in readme (#9260)


--
[...truncated 6.52 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] 

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Sophie Blee-Goldman]

> it's never a good sign when the discussion moves into the vote thread

Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads strikes
again.
Thanks for redirecting me Bruno

I suppose it's unfair to expect the callers to keep perfect track of the
current
 number of stream threads, but it also seems like you shouldn't be calling
removeStreamThread() when there are no threads left. Either you're just
haphazardly removing threads and could unintentionally slip into a state of
no
running threads without realizing it, or more realistically, you're
carefully
removing threads based on some metric(s) that convey whether the system is
over or under-provisioned. If your metrics say you're over-provisioned but
there's
not one thread running, well, that certainly sounds exceptional to me. Or
you might
be right in that the cluster is over-provisioned but have just been
directing the
removeStreamThread() and addStreamThread() calls to instances at random, and
end up with one massive instance and one with no threads at all. Again,
this
probably merits some human intervention (or system redesign)

That said, I don't think there's any real harm to just returning null in
this case, but I hope
that users would pay attention to this since it seems likely to indicate
something has gone
seriously wrong. I suppose Optional would be a reasonable
compromise.

As for the method name, what about activeStreamThreads() or
liveStreamThreads() ?

On Mon, Sep 7, 2020 at 1:45 AM Bruno Cadonna  wrote:

> Hi John,
>
> I agree with you except for checking null. I would rather prefer to use
> Optional as the return type to both methods.
>
> I changed the subject from [VOTE] to [DISCUSS] so that we can follow up
> in the discussion thread.
>
> Best,
> Bruno
>
> On 04.09.20 23:12, John Roesler wrote:
> > Hi Sophie,
> >
> > Uh, oh, it's never a good sign when the discussion moves
> > into the vote thread :)
> >
> > I agree with you, it seems like a good touch for
> > removeStreamThread() to return the name of the thread that
> > got removed, rather than a boolean flag. Maybe the return
> > value would be `null` if there is no thread to remove.
> >
> > If we go that way, I'd suggest that addStreamThread() also
> > return the name of the newly created thread, or null if no
> > thread can be created right now.
> >
> > I'm not completely sure if I think that callers of this
> > method would know exactly how many threads there are. Sure,
> > if a human being is sitting there looking at the metrics or
> > logs and decides to call the method, it would work out, but
> > I'd expect this kind of method to find its way into
> > automated tooling that reacts to things like current system
> > load or resource saturation. Those kinds of toolchains often
> > are part of a distributed system, and it's probably not that
> > easy to guarantee that the thread count they observe is
> > fully consistent with the number of threads that are
> > actually running. Therefore, an in-situ `int
> > numStreamThreads()` method might not be a bad idea. Then
> > again, it seems sort of optional. A caller can catch an
> > exception or react to a `null` return value just the same
> > either way. Having both add/remove methods behave similarly
> > is probably more valuable.
> >
> > Thanks,
> > -John
> >
> >
> > On Thu, 2020-09-03 at 12:15 -0700, Sophie Blee-Goldman
> > wrote:
> >> Hey, sorry for the late reply, I just have one minor suggestion. Since
> we
> >> don't
> >> make any guarantees about which thread gets removed or allow the user to
> >> specify, I think we should return either the index or full name of the
> >> thread
> >> that does get removed by removeThread().
> >>
> >> I know you just updated the KIP to return true/false if there
> are/aren't any
> >> threads to be removed, but I think this would be more appropriate as an
> >> exception than as a return type. I think it's reasonable to expect
> users to
> >> have some sense to how many threads are remaining, and not try to remove
> >> a thread when there is none left. To me, that indicates something wrong
> >> with the user application code and should be treated as an exceptional
> case.
> >> I don't think the same code clarify argument applies here as to the
> >> addStreamThread() case, as there's no reason for an application to be
> >> looping and retrying removeStreamThread()  since if that fails, it's
> because
> >> there are no threads left and thus it will continue to always fail. And
> if
> >> the
> >> user actually wants to shut down all threads, they should just close the
> >> whole application rather than call removeStreamThread() in a loop.
> >>
> >> While I generally think it should be straightforward for users to track
> how
> >> many stream threads they have running, maybe it would be nice to add
> >> a small utility method that does this for them. Something like
> >>
> >> // Returns the number of currently alive threads
> >> boolean runningStreamThreads();
> >>
> >> On Thu, Sep 3, 2020 

[VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

[Jason Gustafson]

Hi All,

I'd like to start a vote on KIP-664:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions.
Thanks for all the feedback!

Best,
Jason

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Sophie Blee-Goldman]

Hey Guozhang & Leah,

I want to push back a bit on the assumption that we would fall back on this
config
in the case of an unspecified window size in a Streams serde. I don't think
it should
be a default at all, either in name or in effect. To borrow the rhetorical
question that
John raised earlier: what is the default window size of an application?

Personally, I agree that that doesn't make much sense. Sure, if you only
have a single
windowed operation in your app then you could just specify the window size
by config,
but how is that any more ergonomic than specifying the window size in the
Serde's
constructor? If anything, it seems worse to put physical and mental
distance between
the specification and the actual usage of such parameters. What if you add
another
windowed operation later, with a different size, and forget to specify the
new size in
the new Serde? Or what if you never specify a default window size config at
all and
accidentally end up using the default config value of Long.MAX_VALUE?
Avoiding this
possibility was/is one of the main motivations of this KIP, and the whole
point of
deprecating the TimeWindowedSerde(innerClass) constructor.

I actually would have advocated to remove this config entirely, but as John
pointed
out, we still need it to configure things like the console consumer

On Tue, Sep 8, 2020 at 10:40 AM Leah Thomas  wrote:

> Hi Guozhang,
>
> Yes, the config would read them as a single window size. I think this
> relates to John's comments about having variably sized windows, which this
> config doesn't handle. I like the name change and updated the wiki to
> reflect that, and to clarify that the default value will still be
> Long.MAX_VALUE.
>
> Thanks for your feedback!
> Leah
>
> On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang  wrote:
>
> > Hello Leah,
> >
> > Thanks for initiating this. I just have one minor clarification question
> > here: the config "window.size.ms" seems to be used as the default window
> > size when reading from a topic that represents windowed records right?
> I.e.
> > if there are multiple topics that represent windowed records but their
> > window sizes are different, with this config we can only read them with a
> > single window size? If yes, could we rename the config as "
> > default.window.size.ms" and make that clear in the description as well?
> > Also we'd better also include its default value which I think would still
> > be MAX_VALUE for compatibility.
> >
> >
> > Guozhang
> >
> >
> > On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas  wrote:
> >
> > > Hey all,
> > >
> > > We should be good to wrap up voting now that the discussion has been
> > > resolved.
> > >
> > > Cheers,
> > > Leah
> > >
> > > On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax 
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On 8/26/20 8:02 AM, John Roesler wrote:
> > > > > Hi all,
> > > > >
> > > > > I've just sent a new message to the DISCUSS thread. We
> > > > > forgot to include the Scala API in the proposal.
> > > > >
> > > > > Thanks,
> > > > > -John
> > > > >
> > > > > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > > > > wrote:
> > > > >> Thanks for the KIP! +1 (non-binding)
> > > > >>
> > > > >> Sophie
> > > > >>
> > > > >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler  >
> > > > wrote:
> > > > >>
> > > > >>> Thanks Leah,
> > > > >>> I’m +1 (binding)
> > > > >>>
> > > > >>> -John
> > > > >>>
> > > > >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
> > > >  Hi everyone,
> > > > 
> > > >  I'd like to kick-off the vote for KIP-659: Improve
> > > >  TimeWindowedDeserializer
> > > >  and TimeWindowedSerde to handle window size.
> > > > 
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
> > > >  Thanks,
> > > >  Leah
> > > > 
> > > > >
> > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-667: Remove deprecated methods from ReadOnlyWindowStore

[Jorge Esteban Quilcate Otoya]

Thanks John! KIP is fixed now.

On Tue, Sep 8, 2020 at 3:45 PM John Roesler  wrote:

> Hello Jorge,
>
> Thanks for the KIP!
>
> FYI, it looks like there's still a boilerplate paragraph in
> the KIP document about it being a template.
>
> The KIP looks good to me; I'm +1 (binding) for doing this in
> 3.0.
>
> Thanks,
> -John
>
> On Mon, 2020-09-07 at 12:18 +0100, Jorge Esteban Quilcate
> Otoya wrote:
> > Hi everyone,
> >
> > I'd like to start a thread to vote for KIP-667 and fix deprecation notes
> on
> > ReadOnlyWindowStore:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-667%3A+Remove+deprecated+methods+from+ReadOnlyWindowStore
> >
> > Discussion thread:
> >
> https://lists.apache.org/thread.html/r499a1b0d9c79c8f38caae2e47db14829878f902e0ee19c1edfdeb819%40%3Cdev.kafka.apache.org%3E
> >
> > Thanks!
> > Jorge.
>
>

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Guozhang Wang]

Thanks Leah, I'm +1 on the KIP proposal.


Guozhang

On Tue, Sep 8, 2020 at 10:40 AM Leah Thomas  wrote:

> Hi Guozhang,
>
> Yes, the config would read them as a single window size. I think this
> relates to John's comments about having variably sized windows, which this
> config doesn't handle. I like the name change and updated the wiki to
> reflect that, and to clarify that the default value will still be
> Long.MAX_VALUE.
>
> Thanks for your feedback!
> Leah
>
> On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang  wrote:
>
> > Hello Leah,
> >
> > Thanks for initiating this. I just have one minor clarification question
> > here: the config "window.size.ms" seems to be used as the default window
> > size when reading from a topic that represents windowed records right?
> I.e.
> > if there are multiple topics that represent windowed records but their
> > window sizes are different, with this config we can only read them with a
> > single window size? If yes, could we rename the config as "
> > default.window.size.ms" and make that clear in the description as well?
> > Also we'd better also include its default value which I think would still
> > be MAX_VALUE for compatibility.
> >
> >
> > Guozhang
> >
> >
> > On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas  wrote:
> >
> > > Hey all,
> > >
> > > We should be good to wrap up voting now that the discussion has been
> > > resolved.
> > >
> > > Cheers,
> > > Leah
> > >
> > > On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax 
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On 8/26/20 8:02 AM, John Roesler wrote:
> > > > > Hi all,
> > > > >
> > > > > I've just sent a new message to the DISCUSS thread. We
> > > > > forgot to include the Scala API in the proposal.
> > > > >
> > > > > Thanks,
> > > > > -John
> > > > >
> > > > > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > > > > wrote:
> > > > >> Thanks for the KIP! +1 (non-binding)
> > > > >>
> > > > >> Sophie
> > > > >>
> > > > >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler  >
> > > > wrote:
> > > > >>
> > > > >>> Thanks Leah,
> > > > >>> I’m +1 (binding)
> > > > >>>
> > > > >>> -John
> > > > >>>
> > > > >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
> > > >  Hi everyone,
> > > > 
> > > >  I'd like to kick-off the vote for KIP-659: Improve
> > > >  TimeWindowedDeserializer
> > > >  and TimeWindowedSerde to handle window size.
> > > > 
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
> > > >  Thanks,
> > > >  Leah
> > > > 
> > > > >
> > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>


-- 
-- Guozhang

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Leah Thomas]

Hi Guozhang,

Yes, the config would read them as a single window size. I think this
relates to John's comments about having variably sized windows, which this
config doesn't handle. I like the name change and updated the wiki to
reflect that, and to clarify that the default value will still be
Long.MAX_VALUE.

Thanks for your feedback!
Leah

On Tue, Sep 8, 2020 at 11:54 AM Guozhang Wang  wrote:

> Hello Leah,
>
> Thanks for initiating this. I just have one minor clarification question
> here: the config "window.size.ms" seems to be used as the default window
> size when reading from a topic that represents windowed records right? I.e.
> if there are multiple topics that represent windowed records but their
> window sizes are different, with this config we can only read them with a
> single window size? If yes, could we rename the config as "
> default.window.size.ms" and make that clear in the description as well?
> Also we'd better also include its default value which I think would still
> be MAX_VALUE for compatibility.
>
>
> Guozhang
>
>
> On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas  wrote:
>
> > Hey all,
> >
> > We should be good to wrap up voting now that the discussion has been
> > resolved.
> >
> > Cheers,
> > Leah
> >
> > On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax  wrote:
> >
> > > +1 (binding)
> > >
> > > On 8/26/20 8:02 AM, John Roesler wrote:
> > > > Hi all,
> > > >
> > > > I've just sent a new message to the DISCUSS thread. We
> > > > forgot to include the Scala API in the proposal.
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > > > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > > > wrote:
> > > >> Thanks for the KIP! +1 (non-binding)
> > > >>
> > > >> Sophie
> > > >>
> > > >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler 
> > > wrote:
> > > >>
> > > >>> Thanks Leah,
> > > >>> I’m +1 (binding)
> > > >>>
> > > >>> -John
> > > >>>
> > > >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
> > >  Hi everyone,
> > > 
> > >  I'd like to kick-off the vote for KIP-659: Improve
> > >  TimeWindowedDeserializer
> > >  and TimeWindowedSerde to handle window size.
> > > 
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
> > >  Thanks,
> > >  Leah
> > > 
> > > >
> > >
> > >
> >
>
>
> --
> -- Guozhang
>

[jira] [Created] (KAFKA-10470) zstd decompression with small batches is slow and causes excessive GC

[Robert Wagner (Jira)]

Robert Wagner created KAFKA-10470:
-

 Summary: zstd decompression with small batches is slow and causes 
excessive GC
 Key: KAFKA-10470
 URL: https://issues.apache.org/jira/browse/KAFKA-10470
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Robert Wagner


Similar to KAFKA-5150 but for zstd instead of LZ4, it appears that a large 
decompression buffer (128kb) created by zstd-jni per batch is causing a 
significant performance bottleneck.

The next upcoming version of zstd-jni (1.4.5-7) will have a new constructor for 
ZstdInputStream that allows the client to pass its own buffer.  A similar fix 
as [PR #2967|https://github.com/apache/kafka/pull/2967] could be used to have 
the  ZstdConstructor use a BufferSupplier to re-use the decompression buffer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Dynamics 365 Inbound and Outbound integrations with Kafka topics

[Stephen Gavosto]

Hi there,

I am doing some research using Kafka topics to support inbound and outbound 
messaging to D365.


>From my research, it appears that you can:

create an Azure Function and retrieve the function URL
parse the the URL to obtain webhook URL
Register new web hook in D365 plugin registration tool (add steps etc)

Does Kafka support the creation of receiver webhooks in a similar manner as 
Azure Functions.



*

For outbound--the action in D365 calls the web hook plugin, then calling the 
webhook URL---would ASB receive that message then pass it to Kafka?

For inbound--can Kafka call the D365 api directly or does it need an Apache 
'app'/service to call and pass the message?

*

EG:

Outbound from D365

D365--->webhook calling Azure Function--->message 
transmitted--->ASB???--->Kafka--->external system

Inbound to D365

D365<---D365 API<---Kafka???<---external system



Stephen Gavosto | Manager, CRM Solution Development
Digital Sales and Service Talent Community
Avanade Inc.
USA, Boston
Cell: 802-230-7828
www.avanade.com
[cid:image003.png@01D685DF.A73C0CA0]

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Guozhang Wang]

Hello Leah,

Thanks for initiating this. I just have one minor clarification question
here: the config "window.size.ms" seems to be used as the default window
size when reading from a topic that represents windowed records right? I.e.
if there are multiple topics that represent windowed records but their
window sizes are different, with this config we can only read them with a
single window size? If yes, could we rename the config as "
default.window.size.ms" and make that clear in the description as well?
Also we'd better also include its default value which I think would still
be MAX_VALUE for compatibility.


Guozhang


On Tue, Sep 8, 2020 at 9:38 AM Leah Thomas  wrote:

> Hey all,
>
> We should be good to wrap up voting now that the discussion has been
> resolved.
>
> Cheers,
> Leah
>
> On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax  wrote:
>
> > +1 (binding)
> >
> > On 8/26/20 8:02 AM, John Roesler wrote:
> > > Hi all,
> > >
> > > I've just sent a new message to the DISCUSS thread. We
> > > forgot to include the Scala API in the proposal.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > > wrote:
> > >> Thanks for the KIP! +1 (non-binding)
> > >>
> > >> Sophie
> > >>
> > >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler 
> > wrote:
> > >>
> > >>> Thanks Leah,
> > >>> I’m +1 (binding)
> > >>>
> > >>> -John
> > >>>
> > >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
> >  Hi everyone,
> > 
> >  I'd like to kick-off the vote for KIP-659: Improve
> >  TimeWindowedDeserializer
> >  and TimeWindowedSerde to handle window size.
> > 
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
> >  Thanks,
> >  Leah
> > 
> > >
> >
> >
>


-- 
-- Guozhang

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

[Leah Thomas]

Hey all,

We should be good to wrap up voting now that the discussion has been
resolved.

Cheers,
Leah

On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax  wrote:

> +1 (binding)
>
> On 8/26/20 8:02 AM, John Roesler wrote:
> > Hi all,
> >
> > I've just sent a new message to the DISCUSS thread. We
> > forgot to include the Scala API in the proposal.
> >
> > Thanks,
> > -John
> >
> > On Mon, 2020-08-24 at 18:00 -0700, Sophie Blee-Goldman
> > wrote:
> >> Thanks for the KIP! +1 (non-binding)
> >>
> >> Sophie
> >>
> >> On Mon, Aug 24, 2020 at 5:06 PM John Roesler 
> wrote:
> >>
> >>> Thanks Leah,
> >>> I’m +1 (binding)
> >>>
> >>> -John
> >>>
> >>> On Mon, Aug 24, 2020, at 16:54, Leah Thomas wrote:
>  Hi everyone,
> 
>  I'd like to kick-off the vote for KIP-659: Improve
>  TimeWindowedDeserializer
>  and TimeWindowedSerde to handle window size.
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-659%3A+Improve+TimeWindowedDeserializer+and+TimeWindowedSerde+to+handle+window+size
>  Thanks,
>  Leah
> 
> >
>
>

[jira] [Created] (KAFKA-10469) describeConfigs() for broker loggers returns incorrect values

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-10469:
---

 Summary: describeConfigs() for broker loggers returns incorrect 
values
 Key: KAFKA-10469
 URL: https://issues.apache.org/jira/browse/KAFKA-10469
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Tom Bentley
Assignee: Tom Bentley


{{Log4jController#loggers}} incorrectly uses the root logger's log level for 
any loggers which lack a configured log level of their own. This is incorrect 
because loggers without an explicit level inherit their level from their parent 
logger and this resolved level might be different from the root logger's level. 
This means that the levels reported from {{Admin.describeConfigs}}, which uses 
{{Log4jController#loggers}} are incorrect. This can be shown by using the 
default {{log4j.properties}} and describing a broker's loggers, it reports

{noformat}
kafka.controller=TRACE
kafka.controller.ControllerChannelManager=INFO
kafka.controller.ControllerEventManager$ControllerEventThread=INFO
kafka.controller.KafkaController=INFO
kafka.controller.RequestSendThread=INFO
kafka.controller.TopicDeletionManager=INFO
kafka.controller.ZkPartitionStateMachine=INFO
kafka.controller.ZkReplicaStateMachine=INFO
{noformat}

The default {{log4j.properties}} does indeed set {{kafka.controller}} to 
{{TRACE}}, but it does not configure the others, so they're actually at 
{{TRACE}} not {{INFO}} as reported.







--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielher]

Hi Jason,

Thank you again for all of your feedback, it is greatly appreciated.
Here are some more changes to the KIP: 
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615=21=17

> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway.

I removed this / static / config in favor of a / dynamic / config 
`supported.configs`. This config is explained in more detail below.

> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.


I added the / dynamic / config `supported.configs` to the KIP. This is a map of 
client software name and version to a list of supported configs registered with 
that software name and version. 

e.g.
{
'ClientInformation(softwareName=apache-kafka-java, 
softwareVersion=x.y.a-SNAPSHOT)': 'acks',
'ClientInformation(softwareName=apache-kafka-java, 
softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence'
}  

The changes propose that `supported.configs` be serialized as a json and stored 
in an entity config alongside dynamic configs and quotas for an entity. This 
config is updated for an entity when producers and consumers register the 
dynamic configurations that they support with the entity. This registration 
happens every time that they request configs.

This makes the information refreshable since `supported.configs` can be deleted 
from an entity. Clients register configs every time they request configs, so 
the compatibility information will eventually be added again and continuously 
updated as new clients associate themselves with the entity.

> 2. Tying dynamic configurations to clientId has some downsides.

The new protocols in the changes to the KIP are based on the protocols for 
client quotas. 
They allow dynamic configs to be tied to a user principal and optionally to a 
client-id without trying to fit multiple entity names and types into the 
{Describe, IncrementalAlter}Configs APIs. They also provide a more expressive 
and extensible interface for dynamic client config entity names and types.

> It is common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.

Would it be reasonable to treat consumer entities slightly differently than 
producer entities since they are different types of clients with different 
needs?
Since the design pattern of this new protocol is built in part for 
extensibility of entity names and types, could consumers be associated with an 
additional entity name and type? 

e.g. Consumer configs could be set at the user, group-id and client-id levels. 
They could be resolved with the following order of precedence from most 
precedent to least precedent.

/config/users//groups//clients/
/config/users//groups/
/config/users/

This doesn’t introduce a new label but it would instead allow dynamic consumer 
configs to be set at the group-id level.

> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.

I updated the KIP and PR so that a JoinGroupRequest will not rebalance the 
group if the session timeout in the JoinGroupRequest is different from the 
session timeout in the GroupCoordinator’s metadata and if the group member that 
is rejoining is in a stable state. In practice this just changes the behavior 
of the leader of the group when the timeout is updated since the leader is the 
only one that will trigger a rebalance when rejoining while in a stable state. 

I would greatly appreciate any more feedback from you about these ideas.

Best,
Ryan

On 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielher]

Hi Jason,

Thank you again for all of your feedback, it is greatly appreciated.
Here are some more changes to the KIP: 
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615=21=17

> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway.

I removed this / static / config in favor of a / dynamic / config 
`supported.configs`. This config is explained in more detail below.

> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.


I added the / dynamic / config `supported.configs` to the KIP. This is a map of 
client software name and version to a list of supported configs registered with 
that software name and version. 

e.g.
{
'ClientInformation(softwareName=apache-kafka-java, 
softwareVersion=x.y.a-SNAPSHOT)': 'acks',
'ClientInformation(softwareName=apache-kafka-java, 
softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence'
}  

The changes propose that `supported.configs` be serialized as a json and stored 
in an entity config alongside dynamic configs and quotas for an entity. This 
config is updated for an entity when producers and consumers register the 
dynamic configurations that they support with the entity. This registration 
happens every time that they request configs.

This makes the information refreshable since `supported.configs` can be deleted 
from an entity. Clients register configs every time they request configs, so 
the compatibility information will eventually be added again and continuously 
updated as new clients associate themselves with the entity.

> 2. Tying dynamic configurations to clientId has some downsides.

The new protocols in the changes to the KIP are based on the protocols for 
client quotas. 
They allow dynamic configs to be tied to a user principal and optionally to a 
client-id without trying to fit multiple entity names and types into the 
{Describe, IncrementalAlter}Configs APIs. They also provide a more expressive 
and extensible interface for dynamic client config entity names and types.

> It is common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.

Would it be reasonable to treat consumer entities slightly differently than 
producer entities since they are different types of clients with different 
needs?
Since the design pattern of this new protocol is built in part for 
extensibility of entity names and types, could consumers be associated with an 
additional entity name and type? 

e.g. Consumer configs could be set at the user, group-id and client-id levels. 
They could be resolved with the following order of precedence from most 
precedent to least precedent.

/config/users//groups//clients/
/config/users//groups/
/config/users/

This doesn’t introduce a new label but it would instead allow dynamic consumer 
configs to be set at the group-id level.

> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.

I updated the KIP and PR so that a JoinGroupRequest will not rebalance the 
group if the session timeout in the JoinGroupRequest is different from the 
session timeout in the GroupCoordinator’s metadata and if the group member that 
is rejoining is in a stable state. In practice this just changes the behavior 
of the leader of the group when the timeout is updated since the leader is the 
only one that will trigger a rebalance when rejoining while in a stable state. 

I would greatly appreciate any more feedback from you about these ideas.

Best,
Ryan

On 

Re: [VOTE] KIP-667: Remove deprecated methods from ReadOnlyWindowStore

[John Roesler]

Hello Jorge,

Thanks for the KIP!

FYI, it looks like there's still a boilerplate paragraph in
the KIP document about it being a template.

The KIP looks good to me; I'm +1 (binding) for doing this in
3.0.

Thanks,
-John

On Mon, 2020-09-07 at 12:18 +0100, Jorge Esteban Quilcate
Otoya wrote:
> Hi everyone,
> 
> I'd like to start a thread to vote for KIP-667 and fix deprecation notes on
> ReadOnlyWindowStore:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-667%3A+Remove+deprecated+methods+from+ReadOnlyWindowStore
> 
> Discussion thread:
> https://lists.apache.org/thread.html/r499a1b0d9c79c8f38caae2e47db14829878f902e0ee19c1edfdeb819%40%3Cdev.kafka.apache.org%3E
> 
> Thanks!
> Jorge.

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Bill Bejeck]

Hi Bruno,

Thanks for letting me know, I've added KIP-662 to the release plan.

-Bill

On Mon, Sep 7, 2020 at 11:33 AM Bruno Cadonna  wrote:

> Hi Bill,
>
> Could you add KIP-662 [1] to the release plan. The KIP has been already
> implemented.
>
> Best,
> Bruno
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-662%3A+Throw+Exception+when+Source+Topics+of+a+Streams+App+are+Deleted
>
> On 26.08.20 16:54, Bill Bejeck wrote:
> > Greetings All!
> >
> > I've published a release plan at
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158872629
> .
> > I have included all of the KIPs that are currently approved, but I'm
> happy
> > to make any adjustments as necessary.
> >
> > The KIP freeze is on September 30 with a target release date of November
> 6.
> >
> > Let me know if there are any objections.
> >
> > Thanks,
> > Bill Bejeck
> >
> > On Fri, Aug 14, 2020 at 4:01 PM John Roesler 
> wrote:
> >
> >> Thanks, Bill!
> >> -John
> >>
> >> On Thu, 2020-08-13 at 15:19 -0700, Ismael Juma wrote:
> >>> Thanks for volunteering Bill. :)
> >>>
> >>> Ismael
> >>>
> >>> On Thu, Aug 13, 2020 at 3:13 PM Bill Bejeck 
> wrote:
> >>>
>  Hi All,
> 
>  I'd like to volunteer to be the release manager for our next feature
>  release, 2.7. If there are no objections, I'll send out the release
> >> plan
>  soon.
> 
>  Thanks,
>  Bill Bejeck
> 
> >>
> >>
> >
>

[jira] [Created] (KAFKA-10468) Log4jController.getLoggers serialization

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-10468:
---

 Summary: Log4jController.getLoggers serialization
 Key: KAFKA-10468
 URL: https://issues.apache.org/jira/browse/KAFKA-10468
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Tom Bentley


{{Log4jController#getLoggers()}} returns a {{java.util.List}} wrapper for a 
Scala {{List}}, which results in a {{ClassNotFoundException}} on any MBean 
client which doesn't have the scala wrapper class on its classpath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP idea: Support of multipart messages

[Ben Stopford]

LinkedIn had something like this. Becket did a talk on it a few years ago.
It would be interesting to know what became of it and if there were lessons
learned.
https://www.youtube.com/watch?v=ZrbaXDYUZY8

On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov <
sibirya...@scrapinghub.com> wrote:

> Hello,
>
> I would like to get your opinions on this KIP idea.
>
> In short it will allow to transfer messages of bigger size than allowed by
> the broker.
>
>
> https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing
>
> If all that makes sense, I'll create a full fledged KIP document and expand
> the idea.
>
> Thanks,
> A.
>


-- 

Ben Stopford

Re: Apply for JIRA contributor authority

[Mickael Maison]

Hi,

I already added you last week and replied to your thread.
See 
https://lists.apache.org/thread.html/r723dfe2906bb52a364bc938726f3be654e50d13e3c6606b3c817c245%40%3Cdev.kafka.apache.org%3E

On Tue, Sep 8, 2020 at 4:16 AM 季伟  wrote:
>
> Hi,
> I want to contribute to Apache Kafka,
> Would you please offer me the contributor permission?
> My JIRA ID is jiweiautohome.
>
> --
>

Re: Request to join Assignee list

[Mickael Maison]

Done. Thanks for your interest in Apache Kafka!

On Tue, Sep 8, 2020 at 5:04 AM Yingshuan Song  wrote:
>
> Hi,
>
> Could you please add me to the JIRA Assignee list. I would like to start
> contributing.
>
> Jira username : ymxz
> Full Name: songyingshuan
>
> Apologies in case I've sent this request to the wrong mailing list.
>
> Thanks,
> Yingshuan Song

[DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

[Viktor Somogyi-Vass]

Hi Everyone,

I'd like to restart the discussion on this. Since the KIP has been revamped
I thought I'd start a new discussion thread.

Link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit

Short summary:
- Would like to introduce a new interface similar to the Authorizer called
Auditor as follows:
public interface Auditor {
audit(Request r, AuthorizableRequestContext c, AclOperation
o, Map isAllowed, Map
errors);
}
- Basically it would pass down the request and the authorization
information to the auditor implementation where various kind of reporting
can be done based on the request.
- A new config would be added called "auditor" which is similar to the
"authorizer" config, but users can pass a list of auditor class names.
- The implementation is expected to be low latency similarly to the
Authorizer.
- A default implementation will be added that logs into a file.

I appreciate any feedback on this.

Best,
Viktor

Re: Backports of KAFKA-9839 onto 2.2, 2.3 and 2.4

[Viktor Somogyi-Vass]

Hi again,

Would someone please have a look at these?

Thanks,
Viktor

On Fri, Aug 28, 2020 at 10:11 AM Viktor Somogyi-Vass <
viktorsomo...@gmail.com> wrote:

> Hi folks,
>
> I have backported KAFKA-9839
>  onto 2.2, 2.3 and 2.4
> respectively as we detected it with a customer running 2.2.
> Would anyone please help me review it?
> The PRs are:
> https://github.com/apache/kafka/pull/9150
> https://github.com/apache/kafka/pull/9151
> https://github.com/apache/kafka/pull/9152
>
> Thanks,
> Viktor
>