Build failed in Jenkins: kafka-trunk-jdk11 #768

[Apache Jenkins Server]

See 


Changes:

[matthias]  KIP-476: Add new getAdmin method to KafkaClientSupplier (#7162)

--
[...truncated 2.59 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name PASSED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes STARTED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes PASSED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes and repartition topic name STARTED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Jo

[jira] [Resolved] (KAFKA-8325) Remove from the incomplete set failed. This should be impossible

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8325.

Fix Version/s: 2.3.1
   Resolution: Fixed

> Remove from the incomplete set failed. This should be impossible
> 
>
> Key: KAFKA-8325
> URL: https://issues.apache.org/jira/browse/KAFKA-8325
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 2.1.0, 2.3.0
>Reporter: Mattia Barbon
>Assignee: Bob Barrett
>Priority: Major
> Fix For: 2.3.1
>
>
> I got this error when using the Kafka producer. So far it happened twice, 
> with an interval of about 1 week.
> {{ERROR [2019-05-05 08:43:07,505] 
> org.apache.kafka.clients.producer.internals.Sender: [Producer 
> clientId=, transactionalId=] Uncaught error in kafka 
> producer I/O thread:}}
> {{ ! java.lang.IllegalStateException: Remove from the incomplete set failed. 
> This should be impossible.}}
> {{ ! at 
> org.apache.kafka.clients.producer.internals.IncompleteBatches.remove(IncompleteBatches.java:44)}}
> {{ ! at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.deallocate(RecordAccumulator.java:645)}}
> {{ ! at 
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:717)}}
> {{ ! at 
> org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:365)}}
> {{ ! at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:308)}}
> {{ ! at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:233)}}
> {{ ! at java.lang.Thread.run(Thread.java:748)}}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [VOTE] KIP-440: Extend Connect Converter to support headers

[sapiensy]

Thanks everyone! Closing the vote with +7 (+3 binding). 

On 2019/08/21 13:41:34, Bill Bejeck  wrote: 
> Thanks for the KIP! This looks like a valuable addition.
> 
> +1(binding)
> 
> -Bill
> 
> On Mon, Aug 5, 2019 at 6:15 PM Ryanne Dolan  wrote:
> 
> > +1, non-binding
> >
> > Ryanne
> >
> > On Mon, Aug 5, 2019 at 3:38 PM Randall Hauch  wrote:
> >
> > > If my math is right, we have 3 non-binding +1 votes and 2 binding +1
> > votes.
> > >
> > > This is a simple but really beneficial KIP for Connect. Can we get
> > another
> > > review and vote by a committer? Thanks!
> > >
> > > Randall
> > >
> > > On Fri, May 31, 2019 at 3:37 PM sapie...@gmail.com 
> > > wrote:
> > >
> > > > Hey everyone, just bumping this thread again. We need one more vote
> > from
> > > > the committers. Thanks! :)
> > > >
> > > > On 2019/05/19 14:31:15, Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > +1 (non-binding). Thanks for the KIP!
> > > > >
> > > > > On Sun, May 19, 2019 at 6:36 PM Dongjin Lee 
> > > wrote:
> > > > >
> > > > > > +1 (non-binding).
> > > > > >
> > > > > > Binding: +2 (Randall, Gwen)
> > > > > > Non-binding: +2 (Andrew, Dongjin)
> > > > > >
> > > > > > We need one more +1 from the committers. Is there anyone else?
> > > > > >
> > > > > > Thanks,
> > > > > > Dongjin
> > > > > >
> > > > > > On Fri, May 10, 2019 at 12:16 AM Andrew Schofield <
> > > > > > andrew_schofi...@live.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding).
> > > > > > >
> > > > > > > Looks good.
> > > > > > >
> > > > > > > On 09/05/2019, 15:55, "Gwen Shapira"  wrote:
> > > > > > >
> > > > > > > +1 (binding)
> > > > > > > Thank you!
> > > > > > >
> > > > > > > On Mon, May 6, 2019, 11:25 PM Yaroslav Tkachenko <
> > > > sapie...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'd like to start a vote for KIP-440: Extend Connect
> > > Converter
> > > > to
> > > > > > > support
> > > > > > > > headers (
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-440%253A%2BExtend%2BConnect%2BConverter%2Bto%2Bsupport%2Bheaders&data=02%7C01%7C%7C82864a9a5f3049e8ca1d08d6d48e5147%7C84df9e7fe9f640afb435%7C1%7C0%7C636930105039335723&sdata=FyQT5pfTwtT2NTMStgSl6zRjiaWFVJqG3%2B4vt5nRYmI%3D&reserved=0
> > > > > > > > )
> > > > > > > >
> > > > > > > > Discussion:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F1fc1e3d2cddd8311d3db7c98f0d09a1a137ca4b20d1f3c8ab203a855%40%253Cdev.kafka.apache.org%253E&data=02%7C01%7C%7C82864a9a5f3049e8ca1d08d6d48e5147%7C84df9e7fe9f640afb435%7C1%7C0%7C636930105039335723&sdata=B2a4Mx9ScWaO3HEEw0LoIRX0ajETAcwUmDxt5Ir5FIs%3D&reserved=0
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Dongjin Lee*
> > > > > > >
> > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > *github:  github.com/dongjinleekr
> > > > > > > linkedin:
> > > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > speakerdeck:
> > > > > > speakerdeck.com/dongjin
> > > > > > > *
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Ron Dagostino]

Thanks, Colin.  The changes you made to the KIP related to the bridge
release help make it clearer.  I still have some confusion about the phrase
"The rolling upgrade from the bridge release will take several steps."
This made me think you are talking about moving from the bridge release to
some other, newer, release that comes after the bridge release.  But I
think what you are getting at is that the bridge release can be run with or
without Zookeeper -- when first upgrading to it Zookeeper remains in use,
but then there is a transition that can be made to engage the warp drive...
I mean the Controller Quorum.  So maybe the phrase should be "The rolling
upgrade through the bridge release -- starting with Zookeeper being in use
and ending with Zookeeper having been replaced by the Controller Quorum --
will take several steps."

Do I understand it correctly, and might some change in phrasing or
additional clarification help others avoid the same confusion I had?

Ron

On Wed, Aug 21, 2019 at 2:31 PM Colin McCabe  wrote:

> On Wed, Aug 21, 2019, at 04:22, Ron Dagostino wrote:
> > Hi Colin.  I like the concept of a "bridge release" for migrating off of
> > Zookeeper, but I worry that it may become a bottleneck if people hesitate
> > to replace Zookeeper -- they would be unable to adopt newer versions of
> > Kafka until taking (what feels to them like) a giant leap.  As an
> example,
> > assuming version 4.0.x of Kafka is the supported bridge release, I  would
> > not be surprised if uptake of the 4.x release and the time-based releases
> > that follow it end up being much slower due to the perceived barrier.
> >
> > Any perceived barrier could be lowered if the 4.0.x release could
> > optionally continue to use Zookeeper -- then the cutover would be two
> > incremental steps (move to 4.0.x, then replace Zookeeper while staying on
> > 4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
> > Zookeeper in one fell swoop).
>
> Hi Ron,
>
> Just to clarify, the "bridge release" will continue to use ZooKeeper.  It
> will not support running without ZooKeeper.  It is the releases that follow
> the bridge release that will remove ZooKeeper.
>
> Also, it's a bit unclear whether the bridge release would be 3.x or 4.x,
> or something to follow.  We do know that the bridge release can't be a 2.x
> release, since it requires at least one incompatible change, removing
> --zookeeper options from all the shell scripts.  (Since we're doing
> semantic versioning, any time we make an incompatible change, we bump the
> major version number.)
>
> In general, using two sources of metadata is a lot more complex and
> error-prone than one.  A lot of the bugs and corner cases we have are the
> result of divergences between the controller and the state in ZooKeeper.
> Eliminating this divergence, and the split-brain scenarios it creates, is a
> major goal of this work.
>
> >
> > Regardless of whether what I wrote above has merit or not, I think the
> KIP
> > should be more explicit about what the upgrade constraints actually are.
> > Can the bridge release be adopted with Zookeeper remaining in place and
> > then cutting over as a second, follow-on step, or must the Controller
> > Quorum nodes be started first and the bridge release cannot be used with
> > Zookeeper at all?
>
> As I mentioned above, the bridge release supports (indeed, requires)
> ZooKeeper.  I have added a little more text about this to KIP-500 which
> hopefully makes it clearer.
>
> best,
> Colin
>
> >  If the bridge release cannot be used with Zookeeper at
> > all, then no version at or beyond the bridge release is available
> > unless/until abandoning Zookeeper; if the bridge release can be used with
> > Zookeeper, then is it the only version that can be used with Zookeeper,
> or
> > can Zookeeper be kept for additional releases if desired?
> >
> > Ron
> >
> > On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino 
> wrote:
> >
> > > Hi Colin.  The diagram up at the top confused me -- specifically, the
> > > lines connecting the controller/active-controller to the brokers.  I
> had
> > > assumed the arrows on those lines represented the direction of data
> flow,
> > > but that is not the case; the arrows actually identify the target of
> the
> > > action, and the non-arrowed end indicates the initiator of the
> action.  For
> > > example, the lines point from the controller to the brokers in the
> "today"
> > > section on the left to show that the controller pushes to the brokers;
> the
> > > lines point from the brokers to the active-controller in the "tomorrow"
> > > section on the right to show that the brokers pull from the
> > > active-controller.  As I said, this confused me because my gut
> instinct was
> > > to interpret the arrow as indicating the direction of data flow, and
> when I
> > > look at the "tomorrow" picture on the right I initially thought
> information
> > > was moving from the brokers to the active-controller.  Did you consider
> > >

Re: KIP Creation permission

[Matthias J. Sax]

Perfect. Done.

On 8/21/19 6:06 PM, Renuka M wrote:
> Yes.. username: rmetukuru
> Email: renumetuk...@gmail.com
> 
> Thanks
> Renuka M
> 
> On Wed, Aug 21, 2019 at 5:58 PM Matthias J. Sax 
> wrote:
> 
>> There is no user with this name.
>>
>> Did you create an account? If yes, we need your wiki id (ie, user name)
>> to grant permissions.
>>
>>
>> -Matthias
>>
>> On 8/21/19 5:51 PM, Renuka M wrote:
>>> renumetuk...@gmail.com
>>>
>>> Thanks
>>> Renuka M
>>>
>>> On Wed, Aug 21, 2019 at 5:28 PM Matthias J. Sax 
>>> wrote:
>>>
 What is your wiki id?

 On 8/21/19 3:46 PM, Renuka M wrote:
> Hi Admin,
>
> Could you please provide me Permissions to create a KIP.
>
> Thanks
> Renuka M
>


>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk8 #3863

[Apache Jenkins Server]

See 


Changes:

[github]  KAFKA-8594: Add version 2.3 to Streams system tests (#7131)

--
[...truncated 5.90 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldCloseProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFeedStoreFromGlobalKTable[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCleanUpPersistentStateStoresOnClose[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 

Re: KIP Creation permission

[Renuka M]

Yes.. username: rmetukuru
Email: renumetuk...@gmail.com

Thanks
Renuka M

On Wed, Aug 21, 2019 at 5:58 PM Matthias J. Sax 
wrote:

> There is no user with this name.
>
> Did you create an account? If yes, we need your wiki id (ie, user name)
> to grant permissions.
>
>
> -Matthias
>
> On 8/21/19 5:51 PM, Renuka M wrote:
> > renumetuk...@gmail.com
> >
> > Thanks
> > Renuka M
> >
> > On Wed, Aug 21, 2019 at 5:28 PM Matthias J. Sax 
> > wrote:
> >
> >> What is your wiki id?
> >>
> >> On 8/21/19 3:46 PM, Renuka M wrote:
> >>> Hi Admin,
> >>>
> >>> Could you please provide me Permissions to create a KIP.
> >>>
> >>> Thanks
> >>> Renuka M
> >>>
> >>
> >>
> >
>
>

Re: KIP Creation permission

[Matthias J. Sax]

There is no user with this name.

Did you create an account? If yes, we need your wiki id (ie, user name)
to grant permissions.


-Matthias

On 8/21/19 5:51 PM, Renuka M wrote:
> renumetuk...@gmail.com
> 
> Thanks
> Renuka M
> 
> On Wed, Aug 21, 2019 at 5:28 PM Matthias J. Sax 
> wrote:
> 
>> What is your wiki id?
>>
>> On 8/21/19 3:46 PM, Renuka M wrote:
>>> Hi Admin,
>>>
>>> Could you please provide me Permissions to create a KIP.
>>>
>>> Thanks
>>> Renuka M
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: KIP Creation permission

[Renuka M]

renumetuk...@gmail.com

Thanks
Renuka M

On Wed, Aug 21, 2019 at 5:28 PM Matthias J. Sax 
wrote:

> What is your wiki id?
>
> On 8/21/19 3:46 PM, Renuka M wrote:
> > Hi Admin,
> >
> > Could you please provide me Permissions to create a KIP.
> >
> > Thanks
> > Renuka M
> >
>
>

Re: KIP Creation permission

[Matthias J. Sax]

What is your wiki id?

On 8/21/19 3:46 PM, Renuka M wrote:
> Hi Admin,
> 
> Could you please provide me Permissions to create a KIP.
> 
> Thanks
> Renuka M
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk11 #767

[Apache Jenkins Server]

See 


Changes:

[github]  KAFKA-8594: Add version 2.3 to Streams system tests (#7131)

--
[...truncated 2.60 MB...]
kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
STARTED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
STARTED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes STARTED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes PASSED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation STARTED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation PASSED

kafka.server.ListOffsetsRequestTest > testResponseIncludesLeaderEpoch STARTED

kafka.server.ListOffsetsRequestTest > testResponseIncludesLeaderEpoch PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
STARTED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides PASSED

kafka.server.BrokerEpochIntegrationTest > 
testControlRequestWithStaleBrokerEpoch STARTED

kafka.server.BrokerEpochIntegrationTest > 
testControlRequestWithStaleBrokerEpoch PASSED

kafka.server.BrokerEpochIntegrationTest > 
testControlRequestWithCorrectBrokerEpoch STARTED

kafka.server.BrokerEpochIntegrationTest > 
testControlRequestWithCorrectBrokerEpoch PASSED

kafka.server.BrokerEpochIntegrationTest > 
testReplicaManagerBrokerEpochMatchesWithZk STARTED

kafka.server.BrokerEpochIntegrationTest > 
testReplicaManagerBrokerEpochMatchesWithZk PASSED

kafka.server.BrokerEpochIntegrationTest > 
testControllerBrokerEpochCacheMatchesWithZk STARTED

kafka.server.BrokerEpochIntegrationTest > 
testControllerBrokerEpochCacheMatchesWithZk PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApi

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Chris Egerton]

Hi all,

I've made some tweaks to the KIP that I believe are improvements. More
detail can be found on the KIP page itself, but as a brief summary, the
three changes are:

- The removal of the internal.request.verification property in favor of
modifying the default value for the connect.protocol property from
"compatible" to "sessioned"
- The renaming of some configurations to use better terminology (mostly
just "request" instead of "key" where appropriate)
- The addition of two new configurations that dictate how session keys are
to be generated

Thanks Ryanne for the feedback so far, hope to hear from some more of you
soon :)

Cheers,

Chris

On Mon, Aug 19, 2019 at 12:02 PM Chris Egerton  wrote:

> Hi Ryanne,
>
> The reasoning for this is provided in the KIP: "There would be no clear
> way to achieve consensus amongst the workers in a cluster on whether to
> switch to this new behavior." To elaborate on this--it would be bad if a
> follower worker began writing task configurations to the topic while the
> leader continued to only listen on the internal REST endpoint. It's
> necessary to ensure that every worker in a cluster supports this new
> behavior before switching it on.
>
> To be fair, it is technically possible to achieve consensus without using
> the group coordination protocol by adding new configurations and using a
> multi-phase rolling upgrade, but the operational complexity of this
> approach would significantly complicate things for the default case of just
> wanting to bump your Connect cluster to the next version without having to
> alter your existing worker config files and with only a single restart per
> worker. The proposed approach provides a much simpler user experience for
> the most common use case and doesn't impose much additional complexity for
> other use cases.
>
> I've updated the KIP to expand on points from your last emails; let me
> know what you think.
>
> Cheers,
>
> Chris
>
> On Thu, Aug 15, 2019 at 4:53 PM Ryanne Dolan 
> wrote:
>
>> Thanks Chris, that makes sense.
>>
>> I know you have already considered this, but I'm not convinced we should
>> rule out using Kafka topics for this purpose. That would enable the same
>> level of security without introducing any new authentication or
>> authorization mechanisms (your keys). And as you say, you'd need to lock
>> down Connect's topics and groups anyway.
>>
>> Can you explain what you mean when you say using Kafka topics would
>> require
>> "reworking the group coordination protocol"? I don't see how these are
>> related. Why would it matter if the workers sent Kafka messages vs POST
>> requests among themselves?
>>
>> Ryanne
>>
>> On Thu, Aug 15, 2019 at 3:57 PM Chris Egerton 
>> wrote:
>>
>> > Hi Ryanne,
>> >
>> > Yes, if the Connect group is left unsecured then that is a potential
>> > vulnerability. However, in a truly secure Connect cluster, the group
>> would
>> > need to be secured anyways to prevent attackers from joining the group
>> with
>> > the intent to either snoop on connector/task configurations or bring the
>> > cluster to a halt by spamming the group with membership requests and
>> then
>> > not running the assigned connectors/tasks. Additionally, for a Connect
>> > cluster to be secure, access to internal topics (for configs, offsets,
>> and
>> > statuses) would also need to be restricted so that attackers could not,
>> > e.g., write arbitrary connector/task configurations to the configs
>> topic.
>> > This is all currently possible in Kafka with the use of ACLs.
>> >
>> > I think the bottom line here is that there's a number of steps that
>> need to
>> > be taken to effectively lock down a Connect cluster; the point of this
>> KIP
>> > is to close a loophole that exists even after all of those steps are
>> taken,
>> > not to completely secure this one vulnerability even when no other
>> security
>> > measures are taken.
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Wed, Aug 14, 2019 at 10:56 PM Ryanne Dolan 
>> > wrote:
>> >
>> > > Chris, I don't understand how the rebalance protocol can be used to
>> give
>> > > out session tokens in a secure way. It seems that any attacker could
>> just
>> > > join the group and sign requests with the provided token. Am I missing
>> > > something?
>> > >
>> > > Ryanne
>> > >
>> > > On Wed, Aug 14, 2019, 2:31 PM Chris Egerton 
>> wrote:
>> > >
>> > > > The KIP page can be found at
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
>> > > > ,
>> > > > by the way. Apologies for neglecting to include it in my initial
>> email!
>> > > >
>> > > > On Wed, Aug 14, 2019 at 12:29 PM Chris Egerton > >
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I'd like to start discussion on a KIP to secure the internal "POST
>> > > > > /connectors//tasks" endpoint for the Connect framework. The
>> > > > proposed
>> > > > > changes address a vulnerability in the framework in its current

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

[Robert Barrett]

+1 (non-binding)

This will be great to have, thanks Jason!

On Wed, Aug 21, 2019 at 4:29 AM Manikumar  wrote:

> +1 (binding).
>
> Thanks for the KIP. LGTM.
>
>
> On Wed, Aug 21, 2019 at 3:12 PM Satish Duggana 
> wrote:
>
> > Hi Jason,
> > +1 (non binding) Thanks for the KIP!
> >
> > Do we need to have a separate JIRA to update the docs as it introduces
> new
> > metrics and a change in behavior for the existing metric?
> >
> >
> >
> > On Wed, Aug 21, 2019 at 2:41 PM Mickael Maison  >
> > wrote:
> >
> > > +1 (non binding)
> > > Thanks Jason
> > >
> > > On Wed, Aug 21, 2019 at 8:15 AM David Jacot 
> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on KIP-352, which is a follow-up to
> KIP-455
> > > to fix
> > > > > a long-known shortcoming of URP reporting and to improve
> reassignment
> > > > > monitoring:
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> > > > > .
> > > > >
> > > > > Note that I have added one new metric following the discussion. It
> > > seemed
> > > > > useful to have a lag metric for reassigning partitions.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > >
> >
>

Re: [VOTE] KIP-360: Improve handling of unknown producer when using EOS

[Jason Gustafson]

Thanks Matthias. With that, I can close this vote. The total is +4 with 3
binding votes.

-Jason

On Wed, Aug 21, 2019 at 3:52 PM Matthias J. Sax 
wrote:

> +1 (binding)
>
> On 4/10/19 12:17 AM, Magnus Edenhill wrote:
> > +1 (non-binding)
> >
> >
> > Den ons 10 apr. 2019 kl 02:38 skrev Guozhang Wang :
> >
> >> +1 (binding). Thanks for the written KIP! The approach lgtm.
> >>
> >> One minor thing: the name of "last epoch" maybe a bit misleading
> (although
> >> it is for internal usage only and will not be exposed to users) for
> future
> >> developers, how about rename it to "required_epoch" and if it is set to
> >> "-1" it means "not required, hence not checks"?
> >>
> >> Guozhang
> >>
> >> On Tue, Apr 9, 2019 at 5:02 PM Jason Gustafson 
> wrote:
> >>
> >>> Bump (for Guozhang)
> >>>
> >>> On Mon, Apr 8, 2019 at 8:55 AM Jason Gustafson 
> >> wrote:
> >>>
>  Hi All,
> 
>  I'd like to start a vote on KIP-360:
> 
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer
>  .
> 
>  +1 from me (duh)
> 
>  Thanks,
>  Jason
> 
> >>>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

KIP Creation permission

[Renuka M]

Hi Admin,

Could you please provide me Permissions to create a KIP.

Thanks
Renuka M

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

[Matthias J. Sax]

Thanks Jason!

LGTM.

On 8/21/19 3:07 PM, Jason Gustafson wrote:
> Hi Matthias,
> 
> Thanks, I appreciate the thorough review. I've revised the section to make
> the logic clearer. I think you have it right except for the 1). We only
> generate a new PID if the epoch cannot be incremented without overflow.
> 
> -Jason
> 
> On Tue, Aug 20, 2019 at 5:45 PM Matthias J. Sax 
> wrote:
> 
>> Thanks for the KIP. I just have some clarification questions to make
>> sure I understand the proposal correctly:
>>
>> 1) "Safe Epoch Incrementing"
>>
>>> When the coordinator receives a new InitProducerId request, we will use
>> the following logic to update the epoch:
>>>
>>> 1. No epoch is provided: the current epoch will be bumped and the last
>> epoch will be set to -1.
>>> 2. Epoch and producerId are provided, and the provided producerId
>> matches the current producerId or the provided producerId matches the
>> previous producerId and the provided epoch is exhausted:
>>>   a. Provided epoch matches current epoch: the last epoch will be
>> set to the current epoch, and the current epoch will be bumped .
>>>   b. Provided epoch matches last epoch: the current epoch will be
>> returned
>>>   c. Else: return INVALID_PRODUCER_EPOCH
>>> 3. Otherwise, return INVALID_PRODUCER_EPOCH
>>
>> Case (1) would be for a new producer. Hence, should we state that "no
>> PID" is provided (instead of "no epoch" is provided?). That might be
>> clearer and it implies that there is no epoch anyway.
>>
>> Case (2) would be for a producer that received `UNKNOWN_PRODUCER_ID`
>> error and tries to re-initialize itself.
>>
>> Case (2a) implies that the producer send its first request and is not
>> fenced. Case (2b) implies that the producer re-tries to re-initialize
>> itself, ie, it first request to re-initilize did not get a respond but
>> was processed by the transaction coordinator. Case (2c) implies that a
>> producer was fenced (similar case 3, even if I am not sure what case 3
>> actually would be?)
>>
>> Please let me know if my understanding is correct.
>>
>> What is still unclear to me is, why case (2 -- or is it only 2b?)
>> requires that the "provide epoch is exhausted"?
>>
>> For case 2b:
>>
>> Assume a producer with PID=100 and epoch=MAX_EPOCH that gets an
>> `UNKNOWN_PRODUCER_ID`. It sends initPidRequest with the corresponding
>> PID/epoch pair. The TC processes the request and creates a new PID=101
>> with new epoch=0, however, the respond to the producer is lost. The TC
>> still stores `currentPid=101`, `currentEpoch=0` and `previousPid=100`,
>> `previoudEpoch=MAX_EPOCH`. When the producer re-tries, the sent
>> PID/epoch still matches the previous PID/epoch pair and hence the TC
>> know it's a retry?
>>
>> If this reasoning is correct, should the logic be as follows:
>>
>> 1. No PID is provided: create a new PID with epoch=0 and set the last
>> epoch to -1.
>> 2. Epoch and producerId are provided
>>a) the provided producerId/epoch matches the current producerId/epoch:
>>   i) if the epoch is not exhausted, bump the epoch
>>   ii) if the epoch is exhausted, create a new PID with epoch=0
>>b) the provided producerId/epoch matches the previous
>> producerId/epoch: respond with current PID/epoch
>>c) Otherwise, return INVALID_PRODUCER_EPOCH
>>
>>
>>
>> -Matthias
>>
>>
>>
>>
>> On 4/4/19 3:47 PM, Jason Gustafson wrote:
>>> Hi Everyone,
>>>
>>> Sorry for the long delay on this KIP. I have updated it to include the
>>> handling of INVALID_PRODUCER_ID_MAPPING as suggested above. If there are
>> no
>>> further comments, I will plan to start a vote early next week.
>>>
>>> Thanks!
>>> Jason
>>>
>>> On Mon, Mar 25, 2019 at 2:33 PM Adam Bellemare >>
>>> wrote:
>>>
 Ach - Sorry. I meant Jason. I had just read a John Roesler email.

 On Mon, Mar 25, 2019 at 5:21 PM Adam Bellemare <
>> adam.bellem...@gmail.com>
 wrote:

> Hi John
>
> What is the status of this KIP?
>
> My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on
> 2.1.1 for a multitude of our internal topics, and I suspect that a
>> proper
> fix is needed.
>
> Adam
>
> On Mon, Jan 7, 2019 at 7:42 PM Guozhang Wang 
>> wrote:
>
>> Thanks Jason. The proposed solution sounds good to me.
>>
>>
>> Guozhang
>>
>> On Mon, Jan 7, 2019 at 3:52 PM Jason Gustafson 
>> wrote:
>>
>>> Hey Guozhang,
>>>
>>> Thanks for sharing the article. The INVALID_PRODUCER_ID_MAPPING error
>>> occurs following expiration of the producerId. It's possible that
>> another
>>> producerId has been installed in its place following expiration (if
>> another
>>> producer instance has become active), or the mapping is empty. We can
>>> safely retry the InitProducerId with the logic in this KIP in order
>> to
>>> detect which case it is. So I'd suggest something like this:
>>>
>>> 1. After receiving INVALID_PRODUCER_ID_MAPPING, t

Re: [VOTE] KIP-360: Improve handling of unknown producer when using EOS

[Matthias J. Sax]

+1 (binding)

On 4/10/19 12:17 AM, Magnus Edenhill wrote:
> +1 (non-binding)
> 
> 
> Den ons 10 apr. 2019 kl 02:38 skrev Guozhang Wang :
> 
>> +1 (binding). Thanks for the written KIP! The approach lgtm.
>>
>> One minor thing: the name of "last epoch" maybe a bit misleading (although
>> it is for internal usage only and will not be exposed to users) for future
>> developers, how about rename it to "required_epoch" and if it is set to
>> "-1" it means "not required, hence not checks"?
>>
>> Guozhang
>>
>> On Tue, Apr 9, 2019 at 5:02 PM Jason Gustafson  wrote:
>>
>>> Bump (for Guozhang)
>>>
>>> On Mon, Apr 8, 2019 at 8:55 AM Jason Gustafson 
>> wrote:
>>>
 Hi All,

 I'd like to start a vote on KIP-360:

>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer
 .

 +1 from me (duh)

 Thanks,
 Jason

>>>
>>
>> --
>> -- Guozhang
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

[Jason Gustafson]

Hi Matthias,

Thanks, I appreciate the thorough review. I've revised the section to make
the logic clearer. I think you have it right except for the 1). We only
generate a new PID if the epoch cannot be incremented without overflow.

-Jason

On Tue, Aug 20, 2019 at 5:45 PM Matthias J. Sax 
wrote:

> Thanks for the KIP. I just have some clarification questions to make
> sure I understand the proposal correctly:
>
> 1) "Safe Epoch Incrementing"
>
> > When the coordinator receives a new InitProducerId request, we will use
> the following logic to update the epoch:
> >
> > 1. No epoch is provided: the current epoch will be bumped and the last
> epoch will be set to -1.
> > 2. Epoch and producerId are provided, and the provided producerId
> matches the current producerId or the provided producerId matches the
> previous producerId and the provided epoch is exhausted:
> >   a. Provided epoch matches current epoch: the last epoch will be
> set to the current epoch, and the current epoch will be bumped .
> >   b. Provided epoch matches last epoch: the current epoch will be
> returned
> >   c. Else: return INVALID_PRODUCER_EPOCH
> > 3. Otherwise, return INVALID_PRODUCER_EPOCH
>
> Case (1) would be for a new producer. Hence, should we state that "no
> PID" is provided (instead of "no epoch" is provided?). That might be
> clearer and it implies that there is no epoch anyway.
>
> Case (2) would be for a producer that received `UNKNOWN_PRODUCER_ID`
> error and tries to re-initialize itself.
>
> Case (2a) implies that the producer send its first request and is not
> fenced. Case (2b) implies that the producer re-tries to re-initialize
> itself, ie, it first request to re-initilize did not get a respond but
> was processed by the transaction coordinator. Case (2c) implies that a
> producer was fenced (similar case 3, even if I am not sure what case 3
> actually would be?)
>
> Please let me know if my understanding is correct.
>
> What is still unclear to me is, why case (2 -- or is it only 2b?)
> requires that the "provide epoch is exhausted"?
>
> For case 2b:
>
> Assume a producer with PID=100 and epoch=MAX_EPOCH that gets an
> `UNKNOWN_PRODUCER_ID`. It sends initPidRequest with the corresponding
> PID/epoch pair. The TC processes the request and creates a new PID=101
> with new epoch=0, however, the respond to the producer is lost. The TC
> still stores `currentPid=101`, `currentEpoch=0` and `previousPid=100`,
> `previoudEpoch=MAX_EPOCH`. When the producer re-tries, the sent
> PID/epoch still matches the previous PID/epoch pair and hence the TC
> know it's a retry?
>
> If this reasoning is correct, should the logic be as follows:
>
> 1. No PID is provided: create a new PID with epoch=0 and set the last
> epoch to -1.
> 2. Epoch and producerId are provided
>a) the provided producerId/epoch matches the current producerId/epoch:
>   i) if the epoch is not exhausted, bump the epoch
>   ii) if the epoch is exhausted, create a new PID with epoch=0
>b) the provided producerId/epoch matches the previous
> producerId/epoch: respond with current PID/epoch
>c) Otherwise, return INVALID_PRODUCER_EPOCH
>
>
>
> -Matthias
>
>
>
>
> On 4/4/19 3:47 PM, Jason Gustafson wrote:
> > Hi Everyone,
> >
> > Sorry for the long delay on this KIP. I have updated it to include the
> > handling of INVALID_PRODUCER_ID_MAPPING as suggested above. If there are
> no
> > further comments, I will plan to start a vote early next week.
> >
> > Thanks!
> > Jason
> >
> > On Mon, Mar 25, 2019 at 2:33 PM Adam Bellemare  >
> > wrote:
> >
> >> Ach - Sorry. I meant Jason. I had just read a John Roesler email.
> >>
> >> On Mon, Mar 25, 2019 at 5:21 PM Adam Bellemare <
> adam.bellem...@gmail.com>
> >> wrote:
> >>
> >>> Hi John
> >>>
> >>> What is the status of this KIP?
> >>>
> >>> My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on
> >>> 2.1.1 for a multitude of our internal topics, and I suspect that a
> proper
> >>> fix is needed.
> >>>
> >>> Adam
> >>>
> >>> On Mon, Jan 7, 2019 at 7:42 PM Guozhang Wang 
> wrote:
> >>>
>  Thanks Jason. The proposed solution sounds good to me.
> 
> 
>  Guozhang
> 
>  On Mon, Jan 7, 2019 at 3:52 PM Jason Gustafson 
>  wrote:
> 
> > Hey Guozhang,
> >
> > Thanks for sharing the article. The INVALID_PRODUCER_ID_MAPPING error
> > occurs following expiration of the producerId. It's possible that
>  another
> > producerId has been installed in its place following expiration (if
>  another
> > producer instance has become active), or the mapping is empty. We can
> > safely retry the InitProducerId with the logic in this KIP in order
> to
> > detect which case it is. So I'd suggest something like this:
> >
> > 1. After receiving INVALID_PRODUCER_ID_MAPPING, the producer can send
> > InitProducerId using the current producerId and epoch.
> > 2. If no mapping exists, the coordinator can generate a new
> 

Re: ACL for group creation?

[Colin McCabe]

I think it's worth considering  separating out the permissions needed to create 
a consumer group from the permissions needed to join one.  We distinguish these 
permissions for topics, and people generally find it useful.  We could start 
checking CREATE on GROUP, perhaps?  It might be hard to do in a compatible way.

cheers,
Colin


On Wed, Aug 21, 2019, at 12:05, Adam Bellemare wrote:
> +users mailing list
> 
> David,
> 
> I don't think I really understand your email. Are you saying that this can
> already be achieved only using the READ ACL?
> 
> Thanks
> Adam
> 
> 
> 
> On Wed, Aug 21, 2019 at 3:58 AM David Jacot  wrote:
> 
> > Hello,
> >
> > It would be better to ask such question on the user mailing list.
> >
> > The reason is that the group is created automatically when a consumer
> > joins it. It is not created explicitly so it can be restricted.
> >
> > In your case, you could setup a ACL to authorize the application to only
> > use the group you have defined. It would prevent the application from
> > creating new groups. (READ Acl on Group resource with a specific name).
> >
> > Best,
> > David
> >
> > On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare 
> > wrote:
> >
> > > Hi All
> > >
> > > I am looking through the Confluent docs and core Kafka docs and don't see
> > > an ACL for group creation:
> > > https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > > and
> > > https://kafka.apache.org/documentation/#security_authz
> > >
> > > My scenario is simple: We use the consumer group as the means of
> > > identifying a single application, including tooling for managing
> > > application resets, offset management, lag monitoring, etc. We often have
> > > situations where someone resets their consumer group by appending an
> > > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > > monitoring and management tooling out of whack.
> > >
> > > Is there a reason why we do not have ACL-based CREATE restrictions to a
> > > particular consumer group? I am willing to do the work to implement this
> > > and test it out, but I wanted to validate that there isn't a reason I am
> > > missing.
> > >
> > > Thanks
> > > Adam
> > >
> >
>

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Colin McCabe]

Hi Ryanne,

Apache Ratis looks like a very interesting project, but I don't think this is 
the right use-case for it.  At its heart, Apache Kafka is a system for managing 
logs.  We should avoid adding a dependency on an external system to manage the 
logs of Kafka itself, since that is one of Kafka's core functions.

In the past we've successfully used Kafka itself to store metadata about 
consumer offsets, transactions, and so on.  This is the same kind of thing, 
except that we need a slightly different replication protocol to avoid the 
dependency that the existing one has on the controller.

I think that down the road, having support for quorum replication in the Kafka 
will be useful for regular topics, not just for the metadata partition.  
Quorum-based replication has dramatically lower tail latencies than the 
acks=all configuration that many people use currently.  The tradeoff is that 
Raft doesn't support redundancy with fewer than 3 replicas.  But that is a 
tradeoff that is appropriate to make for many applications.

best,
Colin

On Wed, Aug 21, 2019, at 12:19, Ryanne Dolan wrote:
> Colin, have you considered leveraging Apache Ratis (incubating)?
> 
> Ryanne
> 
> On Wed, Aug 21, 2019, 1:28 PM Colin McCabe  wrote:
> 
> > On Wed, Aug 21, 2019, at 06:38, Eno Thereska wrote:
> > > Hi Colin,
> > >
> > > Nice KIP! For such a big change it would be good to add a pointer or
> > > two to related work that provides some sort of soft proof that the
> > > approach taken makes sense. Also such work often builds on other work
> > > and it might be useful to trace its roots. May I recommend adding a
> > > pointer to "Tango: Distributed Data Structures over a Shared Log"
> > > (http://www.cs.cornell.edu/~taozou/sosp13/tangososp.pdf)? There are
> > > other papers that are related (e.g., a more recent one one "The
> > > FuzzyLog: A Partially Ordered Shared Log"
> > > (https://www.usenix.org/system/files/osdi18-lockerman.pdf)).
> > >
> > > Both papers would add to the strength of your motivation.
> > >
> >
> > Hi Eno,
> >
> > Good point.  I added a "references" section on the end and added the Tango
> > paper.  I am not sure we need the FuzzyLog one, though.
> >
> > I also added a link to the Raft paper and one of the papers on HDFS, since
> > I feel like these are very relevant here.
> >
> > best,
> > Colin
> >
> > > Cheers
> > > Eno
> > >
> > > On Wed, Aug 21, 2019 at 12:22 PM Ron Dagostino 
> > wrote:
> > > >
> > > > Hi Colin.  I like the concept of a "bridge release" for migrating off
> > of
> > > > Zookeeper, but I worry that it may become a bottleneck if people
> > hesitate
> > > > to replace Zookeeper -- they would be unable to adopt newer versions of
> > > > Kafka until taking (what feels to them like) a giant leap.  As an
> > example,
> > > > assuming version 4.0.x of Kafka is the supported bridge release, I
> > would
> > > > not be surprised if uptake of the 4.x release and the time-based
> > releases
> > > > that follow it end up being much slower due to the perceived barrier.
> > > >
> > > > Any perceived barrier could be lowered if the 4.0.x release could
> > > > optionally continue to use Zookeeper -- then the cutover would be two
> > > > incremental steps (move to 4.0.x, then replace Zookeeper while staying
> > on
> > > > 4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
> > > > Zookeeper in one fell swoop).
> > > >
> > > > Regardless of whether what I wrote above has merit or not, I think the
> > KIP
> > > > should be more explicit about what the upgrade constraints actually
> > are.
> > > > Can the bridge release be adopted with Zookeeper remaining in place and
> > > > then cutting over as a second, follow-on step, or must the Controller
> > > > Quorum nodes be started first and the bridge release cannot be used
> > with
> > > > Zookeeper at all?  If the bridge release cannot be used with Zookeeper
> > at
> > > > all, then no version at or beyond the bridge release is available
> > > > unless/until abandoning Zookeeper; if the bridge release can be used
> > with
> > > > Zookeeper, then is it the only version that can be used with
> > Zookeeper, or
> > > > can Zookeeper be kept for additional releases if desired?
> > > >
> > > > Ron
> > > >
> > > > On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino 
> > wrote:
> > > >
> > > > > Hi Colin.  The diagram up at the top confused me -- specifically, the
> > > > > lines connecting the controller/active-controller to the brokers.  I
> > had
> > > > > assumed the arrows on those lines represented the direction of data
> > flow,
> > > > > but that is not the case; the arrows actually identify the target of
> > the
> > > > > action, and the non-arrowed end indicates the initiator of the
> > action.  For
> > > > > example, the lines point from the controller to the brokers in the
> > "today"
> > > > > section on the left to show that the controller pushes to the
> > brokers; the
> > > > > lines point from the brokers to the active-controller in

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[John Roesler]

Hi Guozhang,

> My impression from your previous email is that inside the algorithm when
we
are "filling" them to instances some deterministic logic would be used to
avoid the above case, is that correct?

Yes, that was my plan, but I didn't formalize it. There was a requirement
that the assignment algorithm must not produce a new assignment if the
current assignment is already balanced, so at the least, any thrashing
would be restricted to the "balancing" phase while tasks are moving around
the cluster.

Anyway, I think it would be good to say that we'll "try to" produce stable
assignments, so I've added a "should" clause to the assignment spec:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-AssignmentAlgorithm

For example, we would sort the stateless tasks and available instances
before assigning them, so that the stateless task assignment would mostly
stay stable between assignments, modulo the compute capacity of the
instances changing a little as active stateful tasks get assigned in more
balanced ways.

Thanks,
-John


On Wed, Aug 21, 2019 at 1:55 PM Guozhang Wang  wrote:

> Hello John,
>
> That sounds reasonable. Just double checked the code that with logging
> disabled the corresponding checkpoint file would not contain any values,
> just like a stateless task. So I think treating them logically the same is
> fine.
>
> Guozhang
>
>
> On Wed, Aug 21, 2019 at 11:41 AM John Roesler  wrote:
>
> > Hi again, Guozhang,
> >
> > While writing up the section on stateless tasks (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statelesstasks
> > ),
> > I reconsidered whether stateful, but non-logged, tasks should actually
> > report a lag of zero, versus not reporting any lag. By the definition of
> > the "StatefulTasksToRankedCandidates" function, the leader would compute
> a
> > lag of zero for these tasks anyway.
> >
> > Therefore, I think the same reasoning that I supplied you for stateless
> > tasks applies, since the member and leader will agree on a lag of zero
> > anyway, we can avoid adding them to the "Task Lags" map, and save some
> > bytes in the JoinGroup request. This would be especially beneficial in an
> > application that uses remote stores for _all_ its state stores, it would
> > have an extremely lightweight JoinGroup request, with no task lags at
> all.
> >
> > WDYT?
> > -John
> >
> > On Wed, Aug 21, 2019 at 1:17 PM John Roesler  wrote:
> >
> > > Thanks, Guozhang.
> > >
> > > (Side note: I noticed on another pass over the discussion that I'd
> missed
> > > addressing your comment about the potential race condition between
> state
> > > cleanup and lag-based assignment. I've added a solution to the
> proposal:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Racebetweenassignmentandstatecleanup
> > > )
> > >
> > > In the JoinGroup (SubscriptionInfo) metadata, stateless tasks are not
> > > represented at all. This should save us some bytes in the request
> > metadata.
> > > If we treated them like non-logged stateful tasks and reported a lag of
> > 0,
> > > the only difference is that the assignor would be able to tell which
> > > members previously hosted that stateless task.
> > >
> > > I'd like to make a simplifying assumption that stateless tasks can just
> > be
> > > freely reassigned with no regard to stickiness at all, without
> impacting
> > > performance. This is almost true. In fact, while assigned a stateless
> > task,
> > > a member fetches batches of records from the broker, so if we move the
> > > stateless task assignment, this buffered input is wasted and just gets
> > > dropped.
> > >
> > > However, we won't be moving the stateless tasks around all the time
> (just
> > > during rebalances), and we have the requirement that the assigment
> > > algorithm must stabilize to guard against perpetually shuffling a
> > stateless
> > > task from one node to another. So, my hope is that this small amount of
> > > inefficiency would not be a performance-dominating factor. In exchange,
> > we
> > > gain the opportunity for the assignment algorithm to use the stateless
> > > tasks as "filler" during unbalanced assignments. For example, if there
> > is a
> > > node that is just warming up with several standby tasks, maybe the
> > > assignment can give more stateless tasks to that node to balance the
> > > computational load across the cluster.
> > >
> > > It's worth noting that such an assignment would still not be considered
> > > "balanced", so the ultimately balanced final state of the assignment
> > (after
> > > task movements) would still have the desired property that each
> stateful
> > > and stateless task is evenly spread across the cluster.
> > >
> > > Does that seem reasonable?
> > >
> > > 

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Ryanne Dolan]

Colin, have you considered leveraging Apache Ratis (incubating)?

Ryanne

On Wed, Aug 21, 2019, 1:28 PM Colin McCabe  wrote:

> On Wed, Aug 21, 2019, at 06:38, Eno Thereska wrote:
> > Hi Colin,
> >
> > Nice KIP! For such a big change it would be good to add a pointer or
> > two to related work that provides some sort of soft proof that the
> > approach taken makes sense. Also such work often builds on other work
> > and it might be useful to trace its roots. May I recommend adding a
> > pointer to "Tango: Distributed Data Structures over a Shared Log"
> > (http://www.cs.cornell.edu/~taozou/sosp13/tangososp.pdf)? There are
> > other papers that are related (e.g., a more recent one one "The
> > FuzzyLog: A Partially Ordered Shared Log"
> > (https://www.usenix.org/system/files/osdi18-lockerman.pdf)).
> >
> > Both papers would add to the strength of your motivation.
> >
>
> Hi Eno,
>
> Good point.  I added a "references" section on the end and added the Tango
> paper.  I am not sure we need the FuzzyLog one, though.
>
> I also added a link to the Raft paper and one of the papers on HDFS, since
> I feel like these are very relevant here.
>
> best,
> Colin
>
> > Cheers
> > Eno
> >
> > On Wed, Aug 21, 2019 at 12:22 PM Ron Dagostino 
> wrote:
> > >
> > > Hi Colin.  I like the concept of a "bridge release" for migrating off
> of
> > > Zookeeper, but I worry that it may become a bottleneck if people
> hesitate
> > > to replace Zookeeper -- they would be unable to adopt newer versions of
> > > Kafka until taking (what feels to them like) a giant leap.  As an
> example,
> > > assuming version 4.0.x of Kafka is the supported bridge release, I
> would
> > > not be surprised if uptake of the 4.x release and the time-based
> releases
> > > that follow it end up being much slower due to the perceived barrier.
> > >
> > > Any perceived barrier could be lowered if the 4.0.x release could
> > > optionally continue to use Zookeeper -- then the cutover would be two
> > > incremental steps (move to 4.0.x, then replace Zookeeper while staying
> on
> > > 4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
> > > Zookeeper in one fell swoop).
> > >
> > > Regardless of whether what I wrote above has merit or not, I think the
> KIP
> > > should be more explicit about what the upgrade constraints actually
> are.
> > > Can the bridge release be adopted with Zookeeper remaining in place and
> > > then cutting over as a second, follow-on step, or must the Controller
> > > Quorum nodes be started first and the bridge release cannot be used
> with
> > > Zookeeper at all?  If the bridge release cannot be used with Zookeeper
> at
> > > all, then no version at or beyond the bridge release is available
> > > unless/until abandoning Zookeeper; if the bridge release can be used
> with
> > > Zookeeper, then is it the only version that can be used with
> Zookeeper, or
> > > can Zookeeper be kept for additional releases if desired?
> > >
> > > Ron
> > >
> > > On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino 
> wrote:
> > >
> > > > Hi Colin.  The diagram up at the top confused me -- specifically, the
> > > > lines connecting the controller/active-controller to the brokers.  I
> had
> > > > assumed the arrows on those lines represented the direction of data
> flow,
> > > > but that is not the case; the arrows actually identify the target of
> the
> > > > action, and the non-arrowed end indicates the initiator of the
> action.  For
> > > > example, the lines point from the controller to the brokers in the
> "today"
> > > > section on the left to show that the controller pushes to the
> brokers; the
> > > > lines point from the brokers to the active-controller in the
> "tomorrow"
> > > > section on the right to show that the brokers pull from the
> > > > active-controller.  As I said, this confused me because my gut
> instinct was
> > > > to interpret the arrow as indicating the direction of data flow, and
> when I
> > > > look at the "tomorrow" picture on the right I initially thought
> information
> > > > was moving from the brokers to the active-controller.  Did you
> consider
> > > > drawing that picture with the arrows reversed in the "tomorrow" side
> so
> > > > that the arrows represent the direction of data flow, and then add
> the
> > > > labels "push" on the "today" side and "pull" on the "tomorrow" side
> to
> > > > indicate who initiates the data flow?  It occurs to me that this
> picture
> > > > may end up being widely distributed, so it might be in everyone's
> interest
> > > > to proactively avoid any possible confusion by being more explicit.
> > > >
> > > > Minor corrections?
> > > > << which
> > > > is partitioned from the active controller
> > > > >>>In the current world, a broker which can contact ZooKeeper but
> which
> > > > is partitioned from the controller
> > > >
> > > > << offline
> > > > >>>Eventually, the active controller will ask the broker to finally
> go
> > > > offline
> > > >
> > > > << to
> > > > the co

Re: ACL for group creation?

[Adam Bellemare]

+users mailing list

David,

I don't think I really understand your email. Are you saying that this can
already be achieved only using the READ ACL?

Thanks
Adam



On Wed, Aug 21, 2019 at 3:58 AM David Jacot  wrote:

> Hello,
>
> It would be better to ask such question on the user mailing list.
>
> The reason is that the group is created automatically when a consumer
> joins it. It is not created explicitly so it can be restricted.
>
> In your case, you could setup a ACL to authorize the application to only
> use the group you have defined. It would prevent the application from
> creating new groups. (READ Acl on Group resource with a specific name).
>
> Best,
> David
>
> On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare 
> wrote:
>
> > Hi All
> >
> > I am looking through the Confluent docs and core Kafka docs and don't see
> > an ACL for group creation:
> > https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > and
> > https://kafka.apache.org/documentation/#security_authz
> >
> > My scenario is simple: We use the consumer group as the means of
> > identifying a single application, including tooling for managing
> > application resets, offset management, lag monitoring, etc. We often have
> > situations where someone resets their consumer group by appending an
> > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > monitoring and management tooling out of whack.
> >
> > Is there a reason why we do not have ACL-based CREATE restrictions to a
> > particular consumer group? I am willing to do the work to implement this
> > and test it out, but I wanted to validate that there isn't a reason I am
> > missing.
> >
> > Thanks
> > Adam
> >
>

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[Guozhang Wang]

Hello John,

That sounds reasonable. Just double checked the code that with logging
disabled the corresponding checkpoint file would not contain any values,
just like a stateless task. So I think treating them logically the same is
fine.

Guozhang


On Wed, Aug 21, 2019 at 11:41 AM John Roesler  wrote:

> Hi again, Guozhang,
>
> While writing up the section on stateless tasks (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statelesstasks
> ),
> I reconsidered whether stateful, but non-logged, tasks should actually
> report a lag of zero, versus not reporting any lag. By the definition of
> the "StatefulTasksToRankedCandidates" function, the leader would compute a
> lag of zero for these tasks anyway.
>
> Therefore, I think the same reasoning that I supplied you for stateless
> tasks applies, since the member and leader will agree on a lag of zero
> anyway, we can avoid adding them to the "Task Lags" map, and save some
> bytes in the JoinGroup request. This would be especially beneficial in an
> application that uses remote stores for _all_ its state stores, it would
> have an extremely lightweight JoinGroup request, with no task lags at all.
>
> WDYT?
> -John
>
> On Wed, Aug 21, 2019 at 1:17 PM John Roesler  wrote:
>
> > Thanks, Guozhang.
> >
> > (Side note: I noticed on another pass over the discussion that I'd missed
> > addressing your comment about the potential race condition between state
> > cleanup and lag-based assignment. I've added a solution to the proposal:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Racebetweenassignmentandstatecleanup
> > )
> >
> > In the JoinGroup (SubscriptionInfo) metadata, stateless tasks are not
> > represented at all. This should save us some bytes in the request
> metadata.
> > If we treated them like non-logged stateful tasks and reported a lag of
> 0,
> > the only difference is that the assignor would be able to tell which
> > members previously hosted that stateless task.
> >
> > I'd like to make a simplifying assumption that stateless tasks can just
> be
> > freely reassigned with no regard to stickiness at all, without impacting
> > performance. This is almost true. In fact, while assigned a stateless
> task,
> > a member fetches batches of records from the broker, so if we move the
> > stateless task assignment, this buffered input is wasted and just gets
> > dropped.
> >
> > However, we won't be moving the stateless tasks around all the time (just
> > during rebalances), and we have the requirement that the assigment
> > algorithm must stabilize to guard against perpetually shuffling a
> stateless
> > task from one node to another. So, my hope is that this small amount of
> > inefficiency would not be a performance-dominating factor. In exchange,
> we
> > gain the opportunity for the assignment algorithm to use the stateless
> > tasks as "filler" during unbalanced assignments. For example, if there
> is a
> > node that is just warming up with several standby tasks, maybe the
> > assignment can give more stateless tasks to that node to balance the
> > computational load across the cluster.
> >
> > It's worth noting that such an assignment would still not be considered
> > "balanced", so the ultimately balanced final state of the assignment
> (after
> > task movements) would still have the desired property that each stateful
> > and stateless task is evenly spread across the cluster.
> >
> > Does that seem reasonable?
> >
> > Thanks,
> > -John
> >
> > On Wed, Aug 21, 2019 at 11:22 AM Guozhang Wang 
> wrote:
> >
> >> Hello John,
> >>
> >> I've made another pass on the wiki page again, overall LGTM. One meta
> >> comment about the "stateless" tasks: how do we represent them in the
> >> metadata? Are they just treated as stateful tasks with logging disabled,
> >> or
> >> are specially handled? It is not very clear in the description.
> >>
> >>
> >> Guozhang
> >>
> >> On Wed, Aug 21, 2019 at 8:43 AM John Roesler  wrote:
> >>
> >> > I have also specifically called out that the assignment must achieve
> >> both
> >> > "instance" and "task" balance:
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22
> >> >
> >> > I've also addressed the problem of state stores with logging disabled:
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statewithloggingdisabled
> >> >
> >> > I believe this addresses all the concerns that have been raised to
> date.
> >> > Apologies if I've overlooked one of your concerns.
> >> >
> >> > Please give the KIP another read and let me know of any further
> >> thoughts!
> >> > Hopefully, we can start the voting on this KI

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Colin McCabe]

On Tue, Aug 20, 2019, at 07:19, Ron Dagostino wrote:
> Hi Colin.  The diagram up at the top confused me -- specifically, the lines
> connecting the controller/active-controller to the brokers.  I had assumed
> the arrows on those lines represented the direction of data flow, but that
> is not the case; the arrows actually identify the target of the action, and
> the non-arrowed end indicates the initiator of the action.  For example,
> the lines point from the controller to the brokers in the "today" section
> on the left to show that the controller pushes to the brokers; the lines
> point from the brokers to the active-controller in the "tomorrow" section
> on the right to show that the brokers pull from the active-controller.  As
> I said, this confused me because my gut instinct was to interpret the arrow
> as indicating the direction of data flow, and when I look at the "tomorrow"
> picture on the right I initially thought information was moving from the
> brokers to the active-controller.  Did you consider drawing that picture
> with the arrows reversed in the "tomorrow" side so that the arrows
> represent the direction of data flow, and then add the labels "push" on the
> "today" side and "pull" on the "tomorrow" side to indicate who initiates
> the data flow?  It occurs to me that this picture may end up being widely
> distributed, so it might be in everyone's interest to proactively avoid any
> possible confusion by being more explicit.

Hi Ron,

That's an interesting point.  I agree that in the second picture, the direction 
of data flow is opposite the direction in which the RPC goes.  The data flows 
from controller to broker, but the RPC is actually made by the broker to the 
controller.

I think very typical for arrows to represent the RPCs that are made, rather 
than the direction that information is flowing in.  For example, when 
diagramming a TCP handshake, the first arrow is typically drawn from the client 
to the server, even if the larger purpose of the connection is to fetch data 
from the server.  This is sort of a convention for diagrams like this.  I think 
reversing it would probably create more confusion than it would prevent, 
especially because with the arrows reversed on the second picture, it would be 
a lot less apparent how it differs from the first.

> 
> Minor corrections?
> << partitioned from the active controller
> >>>In the current world, a broker which can contact ZooKeeper but which is
> partitioned from the controller
> 
> << >>>Eventually, the active controller will ask the broker to finally go
> offline
> 
> << controller
> >>>New versions of the clients should send these operations directly to the
> active controller
> 
> << instead
> >>>In the post-ZK world, the leader will make an RPC to the active
> controller instead
> 
> << controller.
> >>>For example, the brokers may need to forward their requests to the
> active controller.
> 
> << registrations
> >>>The new (active) controller will monitor ZooKeeper for legacy broker
> node registrations

Thanks.  I fixed the wording here as you suggested.

regards,
Colin


> 
> Ron
> 
> On Mon, Aug 19, 2019 at 6:53 PM Colin McCabe  wrote:
> 
> > Hi all,
> >
> > The KIP has been out for a while, so I'm thinking about calling a vote
> > some time this week.
> >
> > best,
> > Colin
> >
> > On Mon, Aug 19, 2019, at 15:52, Colin McCabe wrote:
> > > On Mon, Aug 19, 2019, at 12:52, David Arthur wrote:
> > > > Thanks for the KIP, Colin. This looks great!
> > > >
> > > > I really like the idea of separating the Controller and Broker JVMs.
> > > >
> > > > As you alluded to above, it might be nice to have a separate
> > > > broker-registration API to avoid overloading the metadata fetch API.
> > > >
> > >
> > > Hi David,
> > >
> > > Thanks for taking a look.
> > >
> > > I removed the sentence about MetadataFetch also serving as the broker
> > > registration API.  I think I agree that we will probably want a
> > > separate RPC to fill this role.  We will have a follow-on KIP that will
> > > go into more detail about metadata propagation and registration in the
> > > post-ZK world.  That KIP will also have a full description of the
> > > registration RPC, etc.  For now, I think the important part for KIP-500
> > > is that the broker registers with the controller quorum.  On
> > > registration, the controller quorum assigns it a new broker epoch,
> > > which can distinguish successive broker incarnations.
> > >
> > > >
> > > > When a broker gets a metadata delta, will it be a sequence of deltas
> > since
> > > > the last update or a cumulative delta since the last update?
> > > >
> > >
> > > It will be a sequence of deltas.  Basically, the broker will be reading
> > > from the metadata log.
> > >
> > > >
> > > > Will we include any kind of integrity check on the deltas to ensure
> > the brokers
> > > > have applied them correctly? Perhaps this will be addressed in one of
> > the
> > > > follow-on KIPs.
> > > >
> > >
> > > In general,

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[Guozhang Wang]

Yes that makes sense to me. I was mainly curious to see how we would avoid
threshing stateless tasks back-and-forth but can guarantee "convergence"
since we do not require any stickiness.

My impression from your previous email is that inside the algorithm when we
are "filling" them to instances some deterministic logic would be used to
avoid the above case, is that correct?


Guozhang


On Wed, Aug 21, 2019 at 11:17 AM John Roesler  wrote:

> Thanks, Guozhang.
>
> (Side note: I noticed on another pass over the discussion that I'd missed
> addressing your comment about the potential race condition between state
> cleanup and lag-based assignment. I've added a solution to the proposal:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Racebetweenassignmentandstatecleanup
> )
>
> In the JoinGroup (SubscriptionInfo) metadata, stateless tasks are not
> represented at all. This should save us some bytes in the request metadata.
> If we treated them like non-logged stateful tasks and reported a lag of 0,
> the only difference is that the assignor would be able to tell which
> members previously hosted that stateless task.
>
> I'd like to make a simplifying assumption that stateless tasks can just be
> freely reassigned with no regard to stickiness at all, without impacting
> performance. This is almost true. In fact, while assigned a stateless task,
> a member fetches batches of records from the broker, so if we move the
> stateless task assignment, this buffered input is wasted and just gets
> dropped.
>
> However, we won't be moving the stateless tasks around all the time (just
> during rebalances), and we have the requirement that the assigment
> algorithm must stabilize to guard against perpetually shuffling a stateless
> task from one node to another. So, my hope is that this small amount of
> inefficiency would not be a performance-dominating factor. In exchange, we
> gain the opportunity for the assignment algorithm to use the stateless
> tasks as "filler" during unbalanced assignments. For example, if there is a
> node that is just warming up with several standby tasks, maybe the
> assignment can give more stateless tasks to that node to balance the
> computational load across the cluster.
>
> It's worth noting that such an assignment would still not be considered
> "balanced", so the ultimately balanced final state of the assignment (after
> task movements) would still have the desired property that each stateful
> and stateless task is evenly spread across the cluster.
>
> Does that seem reasonable?
>
> Thanks,
> -John
>
> On Wed, Aug 21, 2019 at 11:22 AM Guozhang Wang  wrote:
>
> > Hello John,
> >
> > I've made another pass on the wiki page again, overall LGTM. One meta
> > comment about the "stateless" tasks: how do we represent them in the
> > metadata? Are they just treated as stateful tasks with logging disabled,
> or
> > are specially handled? It is not very clear in the description.
> >
> >
> > Guozhang
> >
> > On Wed, Aug 21, 2019 at 8:43 AM John Roesler  wrote:
> >
> > > I have also specifically called out that the assignment must achieve
> both
> > > "instance" and "task" balance:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22
> > >
> > > I've also addressed the problem of state stores with logging disabled:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statewithloggingdisabled
> > >
> > > I believe this addresses all the concerns that have been raised to
> date.
> > > Apologies if I've overlooked one of your concerns.
> > >
> > > Please give the KIP another read and let me know of any further
> thoughts!
> > > Hopefully, we can start the voting on this KIP by the end of the week.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Tue, Aug 20, 2019 at 5:16 PM John Roesler 
> wrote:
> > >
> > > > In response to Bruno's concern #2, I've also added that section to
> the
> > > > "Rejected Alternatives" section.
> > > >
> > > > Additionally, after reviewing some other assignment papers, I've
> > > developed
> > > > the concern that specifying which "phases" the assignment algorithm
> > > should
> > > > have, or indeed the logic of it at all, might be a mistake that
> > > > over-constrains our ability to write an optimal algorithm. Therefore,
> > > I've
> > > > also refactored the KIP to just describe the protocol, and specify
> the
> > > > requirements for the assignment algorithm, but not its exact behavior
> > at
> > > > all.
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > > > On Tue, Aug 20, 2019 at 5:13 PM John Roesler 
> > wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> Thanks for the discussion. I've been considering the idea of giving
> > the
> > > >> "catching up

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[John Roesler]

Hi again, Guozhang,

While writing up the section on stateless tasks (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statelesstasks),
I reconsidered whether stateful, but non-logged, tasks should actually
report a lag of zero, versus not reporting any lag. By the definition of
the "StatefulTasksToRankedCandidates" function, the leader would compute a
lag of zero for these tasks anyway.

Therefore, I think the same reasoning that I supplied you for stateless
tasks applies, since the member and leader will agree on a lag of zero
anyway, we can avoid adding them to the "Task Lags" map, and save some
bytes in the JoinGroup request. This would be especially beneficial in an
application that uses remote stores for _all_ its state stores, it would
have an extremely lightweight JoinGroup request, with no task lags at all.

WDYT?
-John

On Wed, Aug 21, 2019 at 1:17 PM John Roesler  wrote:

> Thanks, Guozhang.
>
> (Side note: I noticed on another pass over the discussion that I'd missed
> addressing your comment about the potential race condition between state
> cleanup and lag-based assignment. I've added a solution to the proposal:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Racebetweenassignmentandstatecleanup
> )
>
> In the JoinGroup (SubscriptionInfo) metadata, stateless tasks are not
> represented at all. This should save us some bytes in the request metadata.
> If we treated them like non-logged stateful tasks and reported a lag of 0,
> the only difference is that the assignor would be able to tell which
> members previously hosted that stateless task.
>
> I'd like to make a simplifying assumption that stateless tasks can just be
> freely reassigned with no regard to stickiness at all, without impacting
> performance. This is almost true. In fact, while assigned a stateless task,
> a member fetches batches of records from the broker, so if we move the
> stateless task assignment, this buffered input is wasted and just gets
> dropped.
>
> However, we won't be moving the stateless tasks around all the time (just
> during rebalances), and we have the requirement that the assigment
> algorithm must stabilize to guard against perpetually shuffling a stateless
> task from one node to another. So, my hope is that this small amount of
> inefficiency would not be a performance-dominating factor. In exchange, we
> gain the opportunity for the assignment algorithm to use the stateless
> tasks as "filler" during unbalanced assignments. For example, if there is a
> node that is just warming up with several standby tasks, maybe the
> assignment can give more stateless tasks to that node to balance the
> computational load across the cluster.
>
> It's worth noting that such an assignment would still not be considered
> "balanced", so the ultimately balanced final state of the assignment (after
> task movements) would still have the desired property that each stateful
> and stateless task is evenly spread across the cluster.
>
> Does that seem reasonable?
>
> Thanks,
> -John
>
> On Wed, Aug 21, 2019 at 11:22 AM Guozhang Wang  wrote:
>
>> Hello John,
>>
>> I've made another pass on the wiki page again, overall LGTM. One meta
>> comment about the "stateless" tasks: how do we represent them in the
>> metadata? Are they just treated as stateful tasks with logging disabled,
>> or
>> are specially handled? It is not very clear in the description.
>>
>>
>> Guozhang
>>
>> On Wed, Aug 21, 2019 at 8:43 AM John Roesler  wrote:
>>
>> > I have also specifically called out that the assignment must achieve
>> both
>> > "instance" and "task" balance:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22
>> >
>> > I've also addressed the problem of state stores with logging disabled:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statewithloggingdisabled
>> >
>> > I believe this addresses all the concerns that have been raised to date.
>> > Apologies if I've overlooked one of your concerns.
>> >
>> > Please give the KIP another read and let me know of any further
>> thoughts!
>> > Hopefully, we can start the voting on this KIP by the end of the week.
>> >
>> > Thanks,
>> > -John
>> >
>> > On Tue, Aug 20, 2019 at 5:16 PM John Roesler  wrote:
>> >
>> > > In response to Bruno's concern #2, I've also added that section to the
>> > > "Rejected Alternatives" section.
>> > >
>> > > Additionally, after reviewing some other assignment papers, I've
>> > developed
>> > > the concern that specifying which "phases" the assignment algorithm
>> > should
>> > > have, or indeed the logic of it at all, might be a mistake that
>> > > over-constrains our abi

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Colin McCabe]

On Wed, Aug 21, 2019, at 04:22, Ron Dagostino wrote:
> Hi Colin.  I like the concept of a "bridge release" for migrating off of
> Zookeeper, but I worry that it may become a bottleneck if people hesitate
> to replace Zookeeper -- they would be unable to adopt newer versions of
> Kafka until taking (what feels to them like) a giant leap.  As an example,
> assuming version 4.0.x of Kafka is the supported bridge release, I  would
> not be surprised if uptake of the 4.x release and the time-based releases
> that follow it end up being much slower due to the perceived barrier.
> 
> Any perceived barrier could be lowered if the 4.0.x release could
> optionally continue to use Zookeeper -- then the cutover would be two
> incremental steps (move to 4.0.x, then replace Zookeeper while staying on
> 4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
> Zookeeper in one fell swoop).

Hi Ron,

Just to clarify, the "bridge release" will continue to use ZooKeeper.  It will 
not support running without ZooKeeper.  It is the releases that follow the 
bridge release that will remove ZooKeeper.

Also, it's a bit unclear whether the bridge release would be 3.x or 4.x, or 
something to follow.  We do know that the bridge release can't be a 2.x 
release, since it requires at least one incompatible change, removing 
--zookeeper options from all the shell scripts.  (Since we're doing semantic 
versioning, any time we make an incompatible change, we bump the major version 
number.)

In general, using two sources of metadata is a lot more complex and error-prone 
than one.  A lot of the bugs and corner cases we have are the result of 
divergences between the controller and the state in ZooKeeper.  Eliminating 
this divergence, and the split-brain scenarios it creates, is a major goal of 
this work.

> 
> Regardless of whether what I wrote above has merit or not, I think the KIP
> should be more explicit about what the upgrade constraints actually are.
> Can the bridge release be adopted with Zookeeper remaining in place and
> then cutting over as a second, follow-on step, or must the Controller
> Quorum nodes be started first and the bridge release cannot be used with
> Zookeeper at all?

As I mentioned above, the bridge release supports (indeed, requires) ZooKeeper. 
 I have added a little more text about this to KIP-500 which hopefully makes it 
clearer.

best,
Colin

>  If the bridge release cannot be used with Zookeeper at
> all, then no version at or beyond the bridge release is available
> unless/until abandoning Zookeeper; if the bridge release can be used with
> Zookeeper, then is it the only version that can be used with Zookeeper, or
> can Zookeeper be kept for additional releases if desired?
> 
> Ron
> 
> On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino  wrote:
> 
> > Hi Colin.  The diagram up at the top confused me -- specifically, the
> > lines connecting the controller/active-controller to the brokers.  I had
> > assumed the arrows on those lines represented the direction of data flow,
> > but that is not the case; the arrows actually identify the target of the
> > action, and the non-arrowed end indicates the initiator of the action.  For
> > example, the lines point from the controller to the brokers in the "today"
> > section on the left to show that the controller pushes to the brokers; the
> > lines point from the brokers to the active-controller in the "tomorrow"
> > section on the right to show that the brokers pull from the
> > active-controller.  As I said, this confused me because my gut instinct was
> > to interpret the arrow as indicating the direction of data flow, and when I
> > look at the "tomorrow" picture on the right I initially thought information
> > was moving from the brokers to the active-controller.  Did you consider
> > drawing that picture with the arrows reversed in the "tomorrow" side so
> > that the arrows represent the direction of data flow, and then add the
> > labels "push" on the "today" side and "pull" on the "tomorrow" side to
> > indicate who initiates the data flow?  It occurs to me that this picture
> > may end up being widely distributed, so it might be in everyone's interest
> > to proactively avoid any possible confusion by being more explicit.
> >
> > Minor corrections?
> > << > is partitioned from the active controller
> > >>>In the current world, a broker which can contact ZooKeeper but which
> > is partitioned from the controller
> >
> > << > >>>Eventually, the active controller will ask the broker to finally go
> > offline
> >
> > << > the controller
> > >>>New versions of the clients should send these operations directly to
> > the active controller
> >
> > << > instead
> > >>>In the post-ZK world, the leader will make an RPC to the active
> > controller instead
> >
> > << > controller.
> > >>>For example, the brokers may need to forward their requests to the
> > active controller.
> >
> > << > registrations
> > >>>The new (active) controller will monitor

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Colin McCabe]

On Wed, Aug 21, 2019, at 06:38, Eno Thereska wrote:
> Hi Colin,
> 
> Nice KIP! For such a big change it would be good to add a pointer or
> two to related work that provides some sort of soft proof that the
> approach taken makes sense. Also such work often builds on other work
> and it might be useful to trace its roots. May I recommend adding a
> pointer to "Tango: Distributed Data Structures over a Shared Log"
> (http://www.cs.cornell.edu/~taozou/sosp13/tangososp.pdf)? There are
> other papers that are related (e.g., a more recent one one "The
> FuzzyLog: A Partially Ordered Shared Log"
> (https://www.usenix.org/system/files/osdi18-lockerman.pdf)).
> 
> Both papers would add to the strength of your motivation.
> 

Hi Eno,

Good point.  I added a "references" section on the end and added the Tango 
paper.  I am not sure we need the FuzzyLog one, though.

I also added a link to the Raft paper and one of the papers on HDFS, since I 
feel like these are very relevant here.

best,
Colin

> Cheers
> Eno
> 
> On Wed, Aug 21, 2019 at 12:22 PM Ron Dagostino  wrote:
> >
> > Hi Colin.  I like the concept of a "bridge release" for migrating off of
> > Zookeeper, but I worry that it may become a bottleneck if people hesitate
> > to replace Zookeeper -- they would be unable to adopt newer versions of
> > Kafka until taking (what feels to them like) a giant leap.  As an example,
> > assuming version 4.0.x of Kafka is the supported bridge release, I  would
> > not be surprised if uptake of the 4.x release and the time-based releases
> > that follow it end up being much slower due to the perceived barrier.
> >
> > Any perceived barrier could be lowered if the 4.0.x release could
> > optionally continue to use Zookeeper -- then the cutover would be two
> > incremental steps (move to 4.0.x, then replace Zookeeper while staying on
> > 4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
> > Zookeeper in one fell swoop).
> >
> > Regardless of whether what I wrote above has merit or not, I think the KIP
> > should be more explicit about what the upgrade constraints actually are.
> > Can the bridge release be adopted with Zookeeper remaining in place and
> > then cutting over as a second, follow-on step, or must the Controller
> > Quorum nodes be started first and the bridge release cannot be used with
> > Zookeeper at all?  If the bridge release cannot be used with Zookeeper at
> > all, then no version at or beyond the bridge release is available
> > unless/until abandoning Zookeeper; if the bridge release can be used with
> > Zookeeper, then is it the only version that can be used with Zookeeper, or
> > can Zookeeper be kept for additional releases if desired?
> >
> > Ron
> >
> > On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino  wrote:
> >
> > > Hi Colin.  The diagram up at the top confused me -- specifically, the
> > > lines connecting the controller/active-controller to the brokers.  I had
> > > assumed the arrows on those lines represented the direction of data flow,
> > > but that is not the case; the arrows actually identify the target of the
> > > action, and the non-arrowed end indicates the initiator of the action.  
> > > For
> > > example, the lines point from the controller to the brokers in the "today"
> > > section on the left to show that the controller pushes to the brokers; the
> > > lines point from the brokers to the active-controller in the "tomorrow"
> > > section on the right to show that the brokers pull from the
> > > active-controller.  As I said, this confused me because my gut instinct 
> > > was
> > > to interpret the arrow as indicating the direction of data flow, and when 
> > > I
> > > look at the "tomorrow" picture on the right I initially thought 
> > > information
> > > was moving from the brokers to the active-controller.  Did you consider
> > > drawing that picture with the arrows reversed in the "tomorrow" side so
> > > that the arrows represent the direction of data flow, and then add the
> > > labels "push" on the "today" side and "pull" on the "tomorrow" side to
> > > indicate who initiates the data flow?  It occurs to me that this picture
> > > may end up being widely distributed, so it might be in everyone's interest
> > > to proactively avoid any possible confusion by being more explicit.
> > >
> > > Minor corrections?
> > > << > > is partitioned from the active controller
> > > >>>In the current world, a broker which can contact ZooKeeper but which
> > > is partitioned from the controller
> > >
> > > << > > >>>Eventually, the active controller will ask the broker to finally go
> > > offline
> > >
> > > << > > the controller
> > > >>>New versions of the clients should send these operations directly to
> > > the active controller
> > >
> > > << > > instead
> > > >>>In the post-ZK world, the leader will make an RPC to the active
> > > controller instead
> > >
> > > << > > controller.
> > > >>>For example, the brokers may need to forward their requests to the
> > > active

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[John Roesler]

Thanks, Guozhang.

(Side note: I noticed on another pass over the discussion that I'd missed
addressing your comment about the potential race condition between state
cleanup and lag-based assignment. I've added a solution to the proposal:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Racebetweenassignmentandstatecleanup
)

In the JoinGroup (SubscriptionInfo) metadata, stateless tasks are not
represented at all. This should save us some bytes in the request metadata.
If we treated them like non-logged stateful tasks and reported a lag of 0,
the only difference is that the assignor would be able to tell which
members previously hosted that stateless task.

I'd like to make a simplifying assumption that stateless tasks can just be
freely reassigned with no regard to stickiness at all, without impacting
performance. This is almost true. In fact, while assigned a stateless task,
a member fetches batches of records from the broker, so if we move the
stateless task assignment, this buffered input is wasted and just gets
dropped.

However, we won't be moving the stateless tasks around all the time (just
during rebalances), and we have the requirement that the assigment
algorithm must stabilize to guard against perpetually shuffling a stateless
task from one node to another. So, my hope is that this small amount of
inefficiency would not be a performance-dominating factor. In exchange, we
gain the opportunity for the assignment algorithm to use the stateless
tasks as "filler" during unbalanced assignments. For example, if there is a
node that is just warming up with several standby tasks, maybe the
assignment can give more stateless tasks to that node to balance the
computational load across the cluster.

It's worth noting that such an assignment would still not be considered
"balanced", so the ultimately balanced final state of the assignment (after
task movements) would still have the desired property that each stateful
and stateless task is evenly spread across the cluster.

Does that seem reasonable?

Thanks,
-John

On Wed, Aug 21, 2019 at 11:22 AM Guozhang Wang  wrote:

> Hello John,
>
> I've made another pass on the wiki page again, overall LGTM. One meta
> comment about the "stateless" tasks: how do we represent them in the
> metadata? Are they just treated as stateful tasks with logging disabled, or
> are specially handled? It is not very clear in the description.
>
>
> Guozhang
>
> On Wed, Aug 21, 2019 at 8:43 AM John Roesler  wrote:
>
> > I have also specifically called out that the assignment must achieve both
> > "instance" and "task" balance:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22
> >
> > I've also addressed the problem of state stores with logging disabled:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statewithloggingdisabled
> >
> > I believe this addresses all the concerns that have been raised to date.
> > Apologies if I've overlooked one of your concerns.
> >
> > Please give the KIP another read and let me know of any further thoughts!
> > Hopefully, we can start the voting on this KIP by the end of the week.
> >
> > Thanks,
> > -John
> >
> > On Tue, Aug 20, 2019 at 5:16 PM John Roesler  wrote:
> >
> > > In response to Bruno's concern #2, I've also added that section to the
> > > "Rejected Alternatives" section.
> > >
> > > Additionally, after reviewing some other assignment papers, I've
> > developed
> > > the concern that specifying which "phases" the assignment algorithm
> > should
> > > have, or indeed the logic of it at all, might be a mistake that
> > > over-constrains our ability to write an optimal algorithm. Therefore,
> > I've
> > > also refactored the KIP to just describe the protocol, and specify the
> > > requirements for the assignment algorithm, but not its exact behavior
> at
> > > all.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Tue, Aug 20, 2019 at 5:13 PM John Roesler 
> wrote:
> > >
> > >> Hi All,
> > >>
> > >> Thanks for the discussion. I've been considering the idea of giving
> the
> > >> "catching up" tasks a different name/role. I was in favor initially,
> but
> > >> after working though some details, I think it causes some problems,
> > which
> > >> I've written up in the "rejected alternatives" part of the KIP:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Addinganewkindoftask(%22moving%22,%22recovering%22,%22learner%22)fortaskmovements
> > >>
> > >> Please give it a read and let me know what you think.
> > >>
> > >> Thanks,
> > >> -John
> > >>
> > >> On Thu, Aug 8, 2019 at 7:57 PM Guozhang Wang 
> > wrote:
> > >>
> > >>> I think I agree with you

[jira] [Created] (KAFKA-8825) Add option to reset consumer offset by relative time

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-8825:
--

 Summary: Add option to reset consumer offset by relative time
 Key: KAFKA-8825
 URL: https://issues.apache.org/jira/browse/KAFKA-8825
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson


When the consumer first initializes its position or when it encounters an 
offset out of range error, we have two automatic reset policies: earliest and 
latest. With longer retention times, the earliest reset option is often too 
expensive and the latest reset requires more time to build recent context. 
Users can also set an auto reset policy of "none" and handle it themselves, but 
it may be useful to have a configurable option between earliest and latest. So 
a user can reset to one day ago or one hour ago for example. Would this be 
useful?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[Satish Duggana]

Hi David,
Thanks for the KIP. I have a couple of questions.

>> For the Java client, the idea is to define two constants in the code to 
>> store its name and its version. If possible, the version will be set 
>> automatically based on metadata coming from gradle (or the repo itself) to 
>> avoid having to do manual changes.

Did you consider taking version property by loading
“kafka/kafka-version.properties” as a resource while java client is
initialized?  “kafka/kafka-version.properties” is shipped with
kafka-clients jar.

>> kafka.server:type=ClientMetrics,name=ConnectedClients
I assume this metric value will be the total no of clients connected
to a broker irrespective of whether name and version follow the
expected pattern ([-.\w]+) or not.

>> kafka.server:type=ClientMetrics,name=ConnectedClients,clientname=([-.\w]+),clientversion=([-.\w]+)
It seems client name and client version are treated as tags for
`ConnectedClients` metric. If so, you may implement this metric
similar to `BrokerTopicMetrics` with topic tag as mentioned here[1].
When is the metric removed for a specific client-name and
client-version?

1. 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaRequestHandler.scala#L231

Thanks,
Satish.




On Wed, Aug 21, 2019 at 5:33 PM David Jacot  wrote:
>
> Hi all,
>
> I would like to start a discussion for KIP-511:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers
>
> Let me know what you think.
>
> Best,
> David

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[Guozhang Wang]

Hello John,

I've made another pass on the wiki page again, overall LGTM. One meta
comment about the "stateless" tasks: how do we represent them in the
metadata? Are they just treated as stateful tasks with logging disabled, or
are specially handled? It is not very clear in the description.


Guozhang

On Wed, Aug 21, 2019 at 8:43 AM John Roesler  wrote:

> I have also specifically called out that the assignment must achieve both
> "instance" and "task" balance:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22
>
> I've also addressed the problem of state stores with logging disabled:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statewithloggingdisabled
>
> I believe this addresses all the concerns that have been raised to date.
> Apologies if I've overlooked one of your concerns.
>
> Please give the KIP another read and let me know of any further thoughts!
> Hopefully, we can start the voting on this KIP by the end of the week.
>
> Thanks,
> -John
>
> On Tue, Aug 20, 2019 at 5:16 PM John Roesler  wrote:
>
> > In response to Bruno's concern #2, I've also added that section to the
> > "Rejected Alternatives" section.
> >
> > Additionally, after reviewing some other assignment papers, I've
> developed
> > the concern that specifying which "phases" the assignment algorithm
> should
> > have, or indeed the logic of it at all, might be a mistake that
> > over-constrains our ability to write an optimal algorithm. Therefore,
> I've
> > also refactored the KIP to just describe the protocol, and specify the
> > requirements for the assignment algorithm, but not its exact behavior at
> > all.
> >
> > Thanks,
> > -John
> >
> > On Tue, Aug 20, 2019 at 5:13 PM John Roesler  wrote:
> >
> >> Hi All,
> >>
> >> Thanks for the discussion. I've been considering the idea of giving the
> >> "catching up" tasks a different name/role. I was in favor initially, but
> >> after working though some details, I think it causes some problems,
> which
> >> I've written up in the "rejected alternatives" part of the KIP:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Addinganewkindoftask(%22moving%22,%22recovering%22,%22learner%22)fortaskmovements
> >>
> >> Please give it a read and let me know what you think.
> >>
> >> Thanks,
> >> -John
> >>
> >> On Thu, Aug 8, 2019 at 7:57 PM Guozhang Wang 
> wrote:
> >>
> >>> I think I agree with you Sophie. My gut feeling is that 1) it should
> not
> >>> be
> >>> the major concern in assignor's algorithm for standby tasks not
> catching
> >>> up, but rather be tackled in different modules, and 2) a lot of
> >>> optimization can be down at the stream thread itself, like dedicated
> >>> threading and larger batching, or even complicated scheduling
> mechanisms
> >>> between running, restoring and standby tasks. In anyways, I think we
> can
> >>> take this out of the scope of KIP-441 for now.
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Thu, Aug 8, 2019 at 5:48 PM Sophie Blee-Goldman <
> sop...@confluent.io>
> >>> wrote:
> >>>
> >>> > > we may have other ways to not starving the standby tasks, for
> >>> example, by
> >>> > > using dedicate threads for standby tasks or even consider having
> >>> > *higher> priority for standby than active* so that we always try to
> >>> caught
> >>> > up standby
> >>> > > first, then process active
> >>> >
> >>> > This is an interesting idea, but seems likely to get in the way of
> the
> >>> > original idea of this KIP
> >>> > -- if we always process standby tasks first, then if we are assigned
> a
> >>> new
> >>> > standby task we
> >>> > will have to wait for it to catch up completely before processing any
> >>> > active tasks! That's
> >>> > even worse than the situation this KIP is trying to help with, since
> a
> >>> new
> >>> > standby task has to
> >>> > restore from 0 (whereas an active task at least can take over from
> >>> wherever
> >>> > the standby was).
> >>> >
> >>> > During restoration -- while there exist any restoring tasks -- I
> think
> >>> it's
> >>> > reasonable to de-prioritize the
> >>> > standby tasks and just process restoring and active tasks so both can
> >>> make
> >>> > progress. But we should
> >>> > let them catch up afterwards somehow -- maybe we can apply some kind
> of
> >>> > heuristic, like "if we haven't
> >>> > processed standbys for X iterations, or Y milliseconds, do so now."
> >>> >
> >>> > Actually, it might even be beneficial to avoid processing standbys a
> >>> record
> >>> > or two at a time and instead
> >>> > wait for a large enough batch to build up for the RocksDB
> bulk-loading
> >>> > benefits.
> >>> >
> >>> > I think the "use dedicated threads for standby" is the more promising
> >>> end
> >>> > goal, especially 

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[Guozhang Wang]

Hi John,

Thanks for the added section, I agree with your reasoning and I think we
can still use the standby replicas now.

Guozhang


On Tue, Aug 20, 2019 at 3:13 PM John Roesler  wrote:

> Hi All,
>
> Thanks for the discussion. I've been considering the idea of giving the
> "catching up" tasks a different name/role. I was in favor initially, but
> after working though some details, I think it causes some problems, which
> I've written up in the "rejected alternatives" part of the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Addinganewkindoftask(%22moving%22,%22recovering%22,%22learner%22)fortaskmovements
>
> Please give it a read and let me know what you think.
>
> Thanks,
> -John
>
> On Thu, Aug 8, 2019 at 7:57 PM Guozhang Wang  wrote:
>
> > I think I agree with you Sophie. My gut feeling is that 1) it should not
> be
> > the major concern in assignor's algorithm for standby tasks not catching
> > up, but rather be tackled in different modules, and 2) a lot of
> > optimization can be down at the stream thread itself, like dedicated
> > threading and larger batching, or even complicated scheduling mechanisms
> > between running, restoring and standby tasks. In anyways, I think we can
> > take this out of the scope of KIP-441 for now.
> >
> >
> > Guozhang
> >
> >
> > On Thu, Aug 8, 2019 at 5:48 PM Sophie Blee-Goldman 
> > wrote:
> >
> > > > we may have other ways to not starving the standby tasks, for
> example,
> > by
> > > > using dedicate threads for standby tasks or even consider having
> > > *higher> priority for standby than active* so that we always try to
> > caught
> > > up standby
> > > > first, then process active
> > >
> > > This is an interesting idea, but seems likely to get in the way of the
> > > original idea of this KIP
> > > -- if we always process standby tasks first, then if we are assigned a
> > new
> > > standby task we
> > > will have to wait for it to catch up completely before processing any
> > > active tasks! That's
> > > even worse than the situation this KIP is trying to help with, since a
> > new
> > > standby task has to
> > > restore from 0 (whereas an active task at least can take over from
> > wherever
> > > the standby was).
> > >
> > > During restoration -- while there exist any restoring tasks -- I think
> > it's
> > > reasonable to de-prioritize the
> > > standby tasks and just process restoring and active tasks so both can
> > make
> > > progress. But we should
> > > let them catch up afterwards somehow -- maybe we can apply some kind of
> > > heuristic, like "if we haven't
> > > processed standbys for X iterations, or Y milliseconds, do so now."
> > >
> > > Actually, it might even be beneficial to avoid processing standbys a
> > record
> > > or two at a time and instead
> > > wait for a large enough batch to build up for the RocksDB bulk-loading
> > > benefits.
> > >
> > > I think the "use dedicated threads for standby" is the more promising
> end
> > > goal, especially since
> > > if we split restoration into "restoring tasks" then active and standbys
> > > share almost nothing. But
> > > that seems like follow-up work to the current KIP :)
> > >
> > > On Thu, Aug 8, 2019 at 5:31 PM Sophie Blee-Goldman <
> sop...@confluent.io>
> > > wrote:
> > >
> > > > Stateful tasks with logging disabled seem to be an interesting edge
> > case.
> > > > On the one hand,
> > > > for balancing purposes they should be considered stateful since as
> > > > Guozhang pointed out
> > > > they are still "heavy" in IO costs. But for "catching up" purposes,
> ie
> > > > when allocating standby
> > > > tasks that will become active tasks, they should be considered
> > stateless
> > > > as there is so
> > > > meaningful sense of their lag. We should never allocate standby tasks
> > for
> > > > them during the
> > > > first rebalance, but should ensure they are evenly distributed across
> > > > instances. Maybe we
> > > > should split these into a third category -- after we assign all
> > stateful
> > > > tasks with logging, we
> > > > then distribute the set of logging-disabled stateful tasks to improve
> > > > balance, before lastly
> > > > distributing stateless tasks?
> > > >
> > > > This actually leads into what I was just thinking, which is that we
> > > really
> > > > should distinguish the
> > > > "catch-up" standbys from normal standbys as well as distinguishing
> > > > actively processing tasks
> > > > from active tasks that are still in the restore phase. It's somewhat
> > > > awkward that today, some
> > > > active tasks just start processing immediately while others behave
> more
> > > > like standby than active
> > > > tasks for some time, before switching to real active. They first use
> > the
> > > > restoreConsumer, then
> > > > later only the "normal" consumer.
> > > >
> > > > However, this restore period is still distinct from normal standbys
> in
> > a
> > > > lot of ways -

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

[John Roesler]

I have also specifically called out that the assignment must achieve both
"instance" and "task" balance:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Defining%22balance%22

I've also addressed the problem of state stores with logging disabled:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Statewithloggingdisabled

I believe this addresses all the concerns that have been raised to date.
Apologies if I've overlooked one of your concerns.

Please give the KIP another read and let me know of any further thoughts!
Hopefully, we can start the voting on this KIP by the end of the week.

Thanks,
-John

On Tue, Aug 20, 2019 at 5:16 PM John Roesler  wrote:

> In response to Bruno's concern #2, I've also added that section to the
> "Rejected Alternatives" section.
>
> Additionally, after reviewing some other assignment papers, I've developed
> the concern that specifying which "phases" the assignment algorithm should
> have, or indeed the logic of it at all, might be a mistake that
> over-constrains our ability to write an optimal algorithm. Therefore, I've
> also refactored the KIP to just describe the protocol, and specify the
> requirements for the assignment algorithm, but not its exact behavior at
> all.
>
> Thanks,
> -John
>
> On Tue, Aug 20, 2019 at 5:13 PM John Roesler  wrote:
>
>> Hi All,
>>
>> Thanks for the discussion. I've been considering the idea of giving the
>> "catching up" tasks a different name/role. I was in favor initially, but
>> after working though some details, I think it causes some problems, which
>> I've written up in the "rejected alternatives" part of the KIP:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-Addinganewkindoftask(%22moving%22,%22recovering%22,%22learner%22)fortaskmovements
>>
>> Please give it a read and let me know what you think.
>>
>> Thanks,
>> -John
>>
>> On Thu, Aug 8, 2019 at 7:57 PM Guozhang Wang  wrote:
>>
>>> I think I agree with you Sophie. My gut feeling is that 1) it should not
>>> be
>>> the major concern in assignor's algorithm for standby tasks not catching
>>> up, but rather be tackled in different modules, and 2) a lot of
>>> optimization can be down at the stream thread itself, like dedicated
>>> threading and larger batching, or even complicated scheduling mechanisms
>>> between running, restoring and standby tasks. In anyways, I think we can
>>> take this out of the scope of KIP-441 for now.
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Thu, Aug 8, 2019 at 5:48 PM Sophie Blee-Goldman 
>>> wrote:
>>>
>>> > > we may have other ways to not starving the standby tasks, for
>>> example, by
>>> > > using dedicate threads for standby tasks or even consider having
>>> > *higher> priority for standby than active* so that we always try to
>>> caught
>>> > up standby
>>> > > first, then process active
>>> >
>>> > This is an interesting idea, but seems likely to get in the way of the
>>> > original idea of this KIP
>>> > -- if we always process standby tasks first, then if we are assigned a
>>> new
>>> > standby task we
>>> > will have to wait for it to catch up completely before processing any
>>> > active tasks! That's
>>> > even worse than the situation this KIP is trying to help with, since a
>>> new
>>> > standby task has to
>>> > restore from 0 (whereas an active task at least can take over from
>>> wherever
>>> > the standby was).
>>> >
>>> > During restoration -- while there exist any restoring tasks -- I think
>>> it's
>>> > reasonable to de-prioritize the
>>> > standby tasks and just process restoring and active tasks so both can
>>> make
>>> > progress. But we should
>>> > let them catch up afterwards somehow -- maybe we can apply some kind of
>>> > heuristic, like "if we haven't
>>> > processed standbys for X iterations, or Y milliseconds, do so now."
>>> >
>>> > Actually, it might even be beneficial to avoid processing standbys a
>>> record
>>> > or two at a time and instead
>>> > wait for a large enough batch to build up for the RocksDB bulk-loading
>>> > benefits.
>>> >
>>> > I think the "use dedicated threads for standby" is the more promising
>>> end
>>> > goal, especially since
>>> > if we split restoration into "restoring tasks" then active and standbys
>>> > share almost nothing. But
>>> > that seems like follow-up work to the current KIP :)
>>> >
>>> > On Thu, Aug 8, 2019 at 5:31 PM Sophie Blee-Goldman <
>>> sop...@confluent.io>
>>> > wrote:
>>> >
>>> > > Stateful tasks with logging disabled seem to be an interesting edge
>>> case.
>>> > > On the one hand,
>>> > > for balancing purposes they should be considered stateful since as
>>> > > Guozhang pointed out
>>> > > they are still "heavy" in IO costs. But for "catching up" purposes,
>>> ie
>>> > > when allocating standby
>>> > > ta

[DISCUSS] KIP-510: Metrics library upgrade

[Mario Molina]

Hi there,

I've written KIP-510

proposing the upgrade of the metrics library which Kafka is using.

Please, have a look and let me know what you think,
Thanks,
Mario

Re: [VOTE] KIP-440: Extend Connect Converter to support headers

[Bill Bejeck]

Thanks for the KIP! This looks like a valuable addition.

+1(binding)

-Bill

On Mon, Aug 5, 2019 at 6:15 PM Ryanne Dolan  wrote:

> +1, non-binding
>
> Ryanne
>
> On Mon, Aug 5, 2019 at 3:38 PM Randall Hauch  wrote:
>
> > If my math is right, we have 3 non-binding +1 votes and 2 binding +1
> votes.
> >
> > This is a simple but really beneficial KIP for Connect. Can we get
> another
> > review and vote by a committer? Thanks!
> >
> > Randall
> >
> > On Fri, May 31, 2019 at 3:37 PM sapie...@gmail.com 
> > wrote:
> >
> > > Hey everyone, just bumping this thread again. We need one more vote
> from
> > > the committers. Thanks! :)
> > >
> > > On 2019/05/19 14:31:15, Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > > > +1 (non-binding). Thanks for the KIP!
> > > >
> > > > On Sun, May 19, 2019 at 6:36 PM Dongjin Lee 
> > wrote:
> > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > Binding: +2 (Randall, Gwen)
> > > > > Non-binding: +2 (Andrew, Dongjin)
> > > > >
> > > > > We need one more +1 from the committers. Is there anyone else?
> > > > >
> > > > > Thanks,
> > > > > Dongjin
> > > > >
> > > > > On Fri, May 10, 2019 at 12:16 AM Andrew Schofield <
> > > > > andrew_schofi...@live.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding).
> > > > > >
> > > > > > Looks good.
> > > > > >
> > > > > > On 09/05/2019, 15:55, "Gwen Shapira"  wrote:
> > > > > >
> > > > > > +1 (binding)
> > > > > > Thank you!
> > > > > >
> > > > > > On Mon, May 6, 2019, 11:25 PM Yaroslav Tkachenko <
> > > sapie...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to start a vote for KIP-440: Extend Connect
> > Converter
> > > to
> > > > > > support
> > > > > > > headers (
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-440%253A%2BExtend%2BConnect%2BConverter%2Bto%2Bsupport%2Bheaders&data=02%7C01%7C%7C82864a9a5f3049e8ca1d08d6d48e5147%7C84df9e7fe9f640afb435%7C1%7C0%7C636930105039335723&sdata=FyQT5pfTwtT2NTMStgSl6zRjiaWFVJqG3%2B4vt5nRYmI%3D&reserved=0
> > > > > > > )
> > > > > > >
> > > > > > > Discussion:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F1fc1e3d2cddd8311d3db7c98f0d09a1a137ca4b20d1f3c8ab203a855%40%253Cdev.kafka.apache.org%253E&data=02%7C01%7C%7C82864a9a5f3049e8ca1d08d6d48e5147%7C84df9e7fe9f640afb435%7C1%7C0%7C636930105039335723&sdata=B2a4Mx9ScWaO3HEEw0LoIRX0ajETAcwUmDxt5Ir5FIs%3D&reserved=0
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > > *github:  github.com/dongjinleekr
> > > > > > linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > speakerdeck:
> > > > > speakerdeck.com/dongjin
> > > > > > *
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Eno Thereska]

Hi Colin,

Nice KIP! For such a big change it would be good to add a pointer or
two to related work that provides some sort of soft proof that the
approach taken makes sense. Also such work often builds on other work
and it might be useful to trace its roots. May I recommend adding a
pointer to "Tango: Distributed Data Structures over a Shared Log"
(http://www.cs.cornell.edu/~taozou/sosp13/tangososp.pdf)? There are
other papers that are related (e.g., a more recent one one "The
FuzzyLog: A Partially Ordered Shared Log"
(https://www.usenix.org/system/files/osdi18-lockerman.pdf)).

Both papers would add to the strength of your motivation.

Cheers
Eno

On Wed, Aug 21, 2019 at 12:22 PM Ron Dagostino  wrote:
>
> Hi Colin.  I like the concept of a "bridge release" for migrating off of
> Zookeeper, but I worry that it may become a bottleneck if people hesitate
> to replace Zookeeper -- they would be unable to adopt newer versions of
> Kafka until taking (what feels to them like) a giant leap.  As an example,
> assuming version 4.0.x of Kafka is the supported bridge release, I  would
> not be surprised if uptake of the 4.x release and the time-based releases
> that follow it end up being much slower due to the perceived barrier.
>
> Any perceived barrier could be lowered if the 4.0.x release could
> optionally continue to use Zookeeper -- then the cutover would be two
> incremental steps (move to 4.0.x, then replace Zookeeper while staying on
> 4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
> Zookeeper in one fell swoop).
>
> Regardless of whether what I wrote above has merit or not, I think the KIP
> should be more explicit about what the upgrade constraints actually are.
> Can the bridge release be adopted with Zookeeper remaining in place and
> then cutting over as a second, follow-on step, or must the Controller
> Quorum nodes be started first and the bridge release cannot be used with
> Zookeeper at all?  If the bridge release cannot be used with Zookeeper at
> all, then no version at or beyond the bridge release is available
> unless/until abandoning Zookeeper; if the bridge release can be used with
> Zookeeper, then is it the only version that can be used with Zookeeper, or
> can Zookeeper be kept for additional releases if desired?
>
> Ron
>
> On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino  wrote:
>
> > Hi Colin.  The diagram up at the top confused me -- specifically, the
> > lines connecting the controller/active-controller to the brokers.  I had
> > assumed the arrows on those lines represented the direction of data flow,
> > but that is not the case; the arrows actually identify the target of the
> > action, and the non-arrowed end indicates the initiator of the action.  For
> > example, the lines point from the controller to the brokers in the "today"
> > section on the left to show that the controller pushes to the brokers; the
> > lines point from the brokers to the active-controller in the "tomorrow"
> > section on the right to show that the brokers pull from the
> > active-controller.  As I said, this confused me because my gut instinct was
> > to interpret the arrow as indicating the direction of data flow, and when I
> > look at the "tomorrow" picture on the right I initially thought information
> > was moving from the brokers to the active-controller.  Did you consider
> > drawing that picture with the arrows reversed in the "tomorrow" side so
> > that the arrows represent the direction of data flow, and then add the
> > labels "push" on the "today" side and "pull" on the "tomorrow" side to
> > indicate who initiates the data flow?  It occurs to me that this picture
> > may end up being widely distributed, so it might be in everyone's interest
> > to proactively avoid any possible confusion by being more explicit.
> >
> > Minor corrections?
> > << > is partitioned from the active controller
> > >>>In the current world, a broker which can contact ZooKeeper but which
> > is partitioned from the controller
> >
> > << > >>>Eventually, the active controller will ask the broker to finally go
> > offline
> >
> > << > the controller
> > >>>New versions of the clients should send these operations directly to
> > the active controller
> >
> > << > instead
> > >>>In the post-ZK world, the leader will make an RPC to the active
> > controller instead
> >
> > << > controller.
> > >>>For example, the brokers may need to forward their requests to the
> > active controller.
> >
> > << > registrations
> > >>>The new (active) controller will monitor ZooKeeper for legacy broker
> > node registrations
> >
> > Ron
> >
> > On Mon, Aug 19, 2019 at 6:53 PM Colin McCabe  wrote:
> >
> >> Hi all,
> >>
> >> The KIP has been out for a while, so I'm thinking about calling a vote
> >> some time this week.
> >>
> >> best,
> >> Colin
> >>
> >> On Mon, Aug 19, 2019, at 15:52, Colin McCabe wrote:
> >> > On Mon, Aug 19, 2019, at 12:52, David Arthur wrote:
> >> > > Thanks for the KIP, Colin. This looks great!
>

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

[Dongjin Lee]

Congratulations, Mitchel!

Thanks,
Dongjin

On Wed, Aug 21, 2019 at 3:38 AM Mitchell  wrote:

> Closing the Vote.
> 4 Binding, 9 non-binding.  No against.
>
> On Tue, Aug 20, 2019 at 1:35 PM Jason Gustafson 
> wrote:
> >
> > +1 Thanks for the KIP!
> >
> > -Jason
> >
> > On Thu, Aug 15, 2019 at 4:01 AM Jakub Scholz  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Jakub
> > >
> > > On Sat, Aug 10, 2019 at 8:34 PM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > wrote:
> > >
> > > > Awesome KIP, +1 (non-binding)
> > > >
> > > > Thanks,
> > > > Stanislav
> > > >
> > > > On Fri, Aug 9, 2019 at 11:32 PM Colin McCabe 
> wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > cheers,
> > > > > Colin
> > > > >
> > > > > On Fri, Aug 9, 2019, at 09:56, Ron Dagostino wrote:
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > The simplest of KIPs, with perhaps the biggest impact.  Like
> removing
> > > > > > the thorn from the soles of my feet.
> > > > > >
> > > > > > Thanks for doing it.
> > > > > >
> > > > > > > On Aug 9, 2019, at 12:50 PM, Dongjin Lee 
> > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Two binding +1 (Gwen, Harsha) with Six non-binding +1 until
> now.
> > > > > > > We need one more binding +1.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dongjin
> > > > > > >
> > > > > > >> On Sat, 10 Aug 2019 at 12:59 AM Tom Bentley <
> tbent...@redhat.com>
> > > > > wrote:
> > > > > > >>
> > > > > > >> +1 (non-binding). Thanks!
> > > > > > >>
> > > > > > >> On Fri, Aug 9, 2019 at 12:37 PM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> +1 (non-binding) Thanks for the KIP, so useful.
> > > > > > >>>
> > > > > > >>> On Fri, Aug 9, 2019 at 4:42 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > >  +1 (non binding)
> > > > > >  Thanks for the KIP!
> > > > > > 
> > > > > >  On Fri, Aug 9, 2019 at 9:36 AM Andrew Schofield
> > > > > >   wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > On 09/08/2019, 08:39, "Sönke Liebau" <
> > > > soenke.lie...@opencore.com>
> > > > > >  wrote:
> > > > > > >
> > > > > > >+1 (non-binding)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >On Fri, 9 Aug 2019 at 04:45, Harsha Chintalapani <
> > > > > > >> ka...@harsha.io>
> > > > > >  wrote:
> > > > > > >
> > > > > > >> +1  (binding). much needed!!
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Aug 08, 2019 at 6:43 PM, Gwen Shapira <
> > > > > > >> g...@confluent.io
> > > > > > 
> > > > > >  wrote:
> > > > > > >>
> > > > > > >>> +1 (binding) THANK YOU. It would be +100 if I could.
> > > > > > >>>
> > > > > > >>> On Thu, Aug 8, 2019 at 6:37 PM Mitchell <
> mitche...@gmail.com
> > > > > > >>>
> > > > > >  wrote:
> > > > > > >>>
> > > > > > >>> Hello Dev,
> > > > > > >>> After the discussion I would like to start the vote for
> > > > > > >> KIP-499
> > > > > > >>>
> > > > > > >>> The following command line tools will have the
> > > > > >  `--bootstrap-server`
> > > > > > >>> command line argument added: kafka-console-producer.sh,
> > > > > > >>> kafka-consumer-groups.sh, kafka-consumer-perf-test.sh,
> > > > > > >>> kafka-verifiable-consumer.sh,
> kafka-verifiable-producer.sh
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> -Mitch
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Gwen Shapira
> > > > > > >>> Product Manager | Confluent
> > > > > > >>> 650.450.2760 | @gwenshap
> > > > > > >>> Follow us: Twitter | blog
> > > > > > >
> > > > > > >
> > > > > > >--
> > > > > > >Sönke Liebau
> > > > > > >Partner
> > > > > > >Tel. +49 179 7940878
> > > > > > >OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> Wedel
> > > -
> > > > > > >>> Germany
> > > > > > > --
> > > > > > > *Dongjin Lee*
> > > > > > >
> > > > > > > *A hitchhiker in the mathematical world.*
> > > > > > > *github:  github.com/dongjinleekr
> > > > > > > linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > > speakerdeck:
> > > > > speakerdeck.com/dongjin
> > > > > > > *
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Stanislav
> > > >
> > >
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*
*github:  github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[David Jacot]

Hi all,

I would like to start a discussion for KIP-511:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers

Let me know what you think.

Best,
David

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

[Manikumar]

Hi,

+1 (binding).

Thanks for the updated KIP. LGTM.

Thanks,
Manikumar



On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass 
wrote:

> Hi All,
>
> Bumping this, I'd be happy to get some additional feedback and/or votes.
>
> Thanks,
> Viktor
>
> On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
> > Hi All,
> >
> > I'd like to start a vote on this KIP.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
> >
> > To summarize it: the proposed feature would allow users (usually
> > superusers) to create delegation tokens for other users. This is
> especially
> > helpful in Spark where the delegation token created this way can be
> > distributed to workers.
> >
> > I'd be happy to receive any votes or additional feedback.
> >
> > Viktor
> >
>

Re: [DISCUSS] KIP-435: Internal Partition Reassignment Batching

[Viktor Somogyi-Vass]

Hey Folks,

I think I'll open a vote early next week about this if there are no more
feedback.

Thanks,
Viktor

On Fri, Aug 9, 2019 at 5:25 PM Viktor Somogyi-Vass 
wrote:

> Hey Stanislav,
>
> I reiterated on the current algorithm and indeed it would change the order
> of replicas in ZK but wouldn't do a leader election, so one would need to
> run the preferred replica election tool. However still in the light of this
> I'm not sure I'd keep the existing behavior as users wouldn't win anything
> with it. Changing leadership automatically would result in a simpler,
> easier and more responsive reassign algorithm which is especially important
> if the reassignment is done to free up the broker from under heavy load.
> Automated tools (Cruise Control) would also have simpler life.
> Let me know what you think.
>
> Viktor
>
> On Thu, Jul 11, 2019 at 7:16 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
>> Hi Stan,
>>
>> I meant the following (maybe rare) scenario - we have replicas [1, 2, 3]
>> on
>> a lot of partitions and the user runs a massive rebalance to change them
>> all to [3, 2, 1]. In the old behavior, I think that this would not do
>> anything but simply change the replica set in ZK.
>> Then, the user could run kafka-preferred-replica-election.sh on a given
>> set
>> of partitions to make sure the new leader 3 gets elected.
>>
>> I thought the old algorithm would elect 3 as the leader in this case
>> right away at the end but I have to double check this. In any case I think
>> it would make sense in the new algorithm if we elected the new preferred
>> leader right away, regardless of the new leader is chosen from the existing
>> replicas or not. If the whole reassignment is in fact just changing the
>> replica order then either way it is a simple (trivial) operation and doing
>> it batched wouldn't slow it down much as there is no data movement
>> involved. If the reassignment is mixed, meaning it contains reordering and
>> real movement as well then in fact it would help to even out the load
>> faster as data movements can get long. For instance in case of a
>> reassignment batch of two partitions concurrently: P1: (1,2,3) -> (3,2,1)
>> and P2: (4,5,6) -> (7,8,9) the P2 reassignment would elect a new leader but
>> P1 wouldn't and it wouldn't help the goal of normalizing traffic on broker
>> 1 that much.
>> Again, I'll have to check how the current algorithm works and if it has
>> any unknown drawbacks to implement what I sketched up above.
>>
>> As for generic preferred leader election when called from the admin api
>> or the auto leader balance feature I think you're right that we should
>> leave it as it is. It doesn't involve any data movement so it's fairly fast
>> and it normalizes the cluster state quickly.
>>
>> Viktor
>>
>> On Tue, Jul 9, 2019 at 9:04 PM Stanislav Kozlovski <
>> stanis...@confluent.io> wrote:
>>
>>> Hey Viktor,
>>>
>>>  I think it is intuitive if there are on a global level...If we applied
>>> > them on every batch then we
>>> > couldn't really guarantee any limits as the user would be able to get
>>> > around it with submitting lots of reassignments.
>>>
>>>
>>> Agreed. Could we mention this explicitly in the KIP?
>>>
>>> Also if I understand correctly, AlterPartitionAssignmentsRequest would
>>> be a
>>> > partition level batching, isn't it? So if you submit 10 partitions at
>>> once
>>> > then they'll all be started by the controller immediately as per my
>>> > understanding.
>>>
>>>
>>> Yep, absolutely
>>>
>>> I've raised the ordering problem on the discussion of KIP-455 in a bit
>>> > different form and as I remember the verdict there was that we
>>> shouldn't
>>> > expose ordering as an API. It might not be easy as you say and there
>>> might
>>> > be much better strategies to follow (like disk or network utilization
>>> > goals). Therefore I'll remove this section from the KIP.
>>>
>>>
>>> Sounds good to me.
>>>
>>> I'm not sure I get this scenario. So are you saying that after they
>>> > submitted a reassignment they also submit a preferred leader change?
>>> > In my mind what I would do is:
>>> > i) make auto.leader.rebalance.enable to obey the leader movement limit
>>> as
>>> > this way it will be easier to calculate the reassignment batches.
>>> >
>>>
>>> Sorry, this is my fault -- I should have been more clear.
>>> First, I didn't think through this well enough at the time, I don't
>>> think.
>>> If we have replicas=[1, 2, 3] and we reassign them to [4, 5, 6], it is
>>> obvious that a leader shift will happen. Your KIP proposes we shift
>>> replicas 1 and 4 first.
>>>
>>> I meant the following (maybe rare) scenario - we have replicas [1, 2, 3]
>>> on
>>> a lot of partitions and the user runs a massive rebalance to change them
>>> all to [3, 2, 1]. In the old behavior, I think that this would not do
>>> anything but simply change the replica set in ZK.
>>> Then, the user could run kafka-preferred-replica-election.sh on a given
>>> set
>>> of partition

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

[Manikumar]

+1 (binding).

Thanks for the KIP. LGTM.


On Wed, Aug 21, 2019 at 3:12 PM Satish Duggana 
wrote:

> Hi Jason,
> +1 (non binding) Thanks for the KIP!
>
> Do we need to have a separate JIRA to update the docs as it introduces new
> metrics and a change in behavior for the existing metric?
>
>
>
> On Wed, Aug 21, 2019 at 2:41 PM Mickael Maison 
> wrote:
>
> > +1 (non binding)
> > Thanks Jason
> >
> > On Wed, Aug 21, 2019 at 8:15 AM David Jacot  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Thanks for the KIP!
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455
> > to fix
> > > > a long-known shortcoming of URP reporting and to improve reassignment
> > > > monitoring:
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> > > > .
> > > >
> > > > Note that I have added one new metric following the discussion. It
> > seemed
> > > > useful to have a lag metric for reassigning partitions.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> >
>

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Ron Dagostino]

Hi Colin.  I like the concept of a "bridge release" for migrating off of
Zookeeper, but I worry that it may become a bottleneck if people hesitate
to replace Zookeeper -- they would be unable to adopt newer versions of
Kafka until taking (what feels to them like) a giant leap.  As an example,
assuming version 4.0.x of Kafka is the supported bridge release, I  would
not be surprised if uptake of the 4.x release and the time-based releases
that follow it end up being much slower due to the perceived barrier.

Any perceived barrier could be lowered if the 4.0.x release could
optionally continue to use Zookeeper -- then the cutover would be two
incremental steps (move to 4.0.x, then replace Zookeeper while staying on
4.0.x) as opposed to a single big-bang (upgrade to 4.0.x and replace
Zookeeper in one fell swoop).

Regardless of whether what I wrote above has merit or not, I think the KIP
should be more explicit about what the upgrade constraints actually are.
Can the bridge release be adopted with Zookeeper remaining in place and
then cutting over as a second, follow-on step, or must the Controller
Quorum nodes be started first and the bridge release cannot be used with
Zookeeper at all?  If the bridge release cannot be used with Zookeeper at
all, then no version at or beyond the bridge release is available
unless/until abandoning Zookeeper; if the bridge release can be used with
Zookeeper, then is it the only version that can be used with Zookeeper, or
can Zookeeper be kept for additional releases if desired?

Ron

On Tue, Aug 20, 2019 at 10:19 AM Ron Dagostino  wrote:

> Hi Colin.  The diagram up at the top confused me -- specifically, the
> lines connecting the controller/active-controller to the brokers.  I had
> assumed the arrows on those lines represented the direction of data flow,
> but that is not the case; the arrows actually identify the target of the
> action, and the non-arrowed end indicates the initiator of the action.  For
> example, the lines point from the controller to the brokers in the "today"
> section on the left to show that the controller pushes to the brokers; the
> lines point from the brokers to the active-controller in the "tomorrow"
> section on the right to show that the brokers pull from the
> active-controller.  As I said, this confused me because my gut instinct was
> to interpret the arrow as indicating the direction of data flow, and when I
> look at the "tomorrow" picture on the right I initially thought information
> was moving from the brokers to the active-controller.  Did you consider
> drawing that picture with the arrows reversed in the "tomorrow" side so
> that the arrows represent the direction of data flow, and then add the
> labels "push" on the "today" side and "pull" on the "tomorrow" side to
> indicate who initiates the data flow?  It occurs to me that this picture
> may end up being widely distributed, so it might be in everyone's interest
> to proactively avoid any possible confusion by being more explicit.
>
> Minor corrections?
> << is partitioned from the active controller
> >>>In the current world, a broker which can contact ZooKeeper but which
> is partitioned from the controller
>
> << >>>Eventually, the active controller will ask the broker to finally go
> offline
>
> << the controller
> >>>New versions of the clients should send these operations directly to
> the active controller
>
> << instead
> >>>In the post-ZK world, the leader will make an RPC to the active
> controller instead
>
> << controller.
> >>>For example, the brokers may need to forward their requests to the
> active controller.
>
> << registrations
> >>>The new (active) controller will monitor ZooKeeper for legacy broker
> node registrations
>
> Ron
>
> On Mon, Aug 19, 2019 at 6:53 PM Colin McCabe  wrote:
>
>> Hi all,
>>
>> The KIP has been out for a while, so I'm thinking about calling a vote
>> some time this week.
>>
>> best,
>> Colin
>>
>> On Mon, Aug 19, 2019, at 15:52, Colin McCabe wrote:
>> > On Mon, Aug 19, 2019, at 12:52, David Arthur wrote:
>> > > Thanks for the KIP, Colin. This looks great!
>> > >
>> > > I really like the idea of separating the Controller and Broker JVMs.
>> > >
>> > > As you alluded to above, it might be nice to have a separate
>> > > broker-registration API to avoid overloading the metadata fetch API.
>> > >
>> >
>> > Hi David,
>> >
>> > Thanks for taking a look.
>> >
>> > I removed the sentence about MetadataFetch also serving as the broker
>> > registration API.  I think I agree that we will probably want a
>> > separate RPC to fill this role.  We will have a follow-on KIP that will
>> > go into more detail about metadata propagation and registration in the
>> > post-ZK world.  That KIP will also have a full description of the
>> > registration RPC, etc.  For now, I think the important part for KIP-500
>> > is that the broker registers with the controller quorum.  On
>> > registration, the controller quorum assigns it a new broker epoch,
>> > which can

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

[Satish Duggana]

Hi Jason,
+1 (non binding) Thanks for the KIP!

Do we need to have a separate JIRA to update the docs as it introduces new
metrics and a change in behavior for the existing metric?



On Wed, Aug 21, 2019 at 2:41 PM Mickael Maison 
wrote:

> +1 (non binding)
> Thanks Jason
>
> On Wed, Aug 21, 2019 at 8:15 AM David Jacot  wrote:
> >
> > +1 (non-binding)
> >
> > Thanks for the KIP!
> >
> > Best,
> > David
> >
> > On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson 
> wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455
> to fix
> > > a long-known shortcoming of URP reporting and to improve reassignment
> > > monitoring:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> > > .
> > >
> > > Note that I have added one new metric following the discussion. It
> seemed
> > > useful to have a lag metric for reassigning partitions.
> > >
> > > Thanks,
> > > Jason
> > >
>

[jira] [Created] (KAFKA-8824) InMemoryTimeOrderedKeyValueBuffer propagates nulls when supress is configured

[Ferran altimiras (Jira)]

Ferran altimiras created KAFKA-8824:
---

 Summary: InMemoryTimeOrderedKeyValueBuffer propagates nulls when 
supress is configured 
 Key: KAFKA-8824
 URL: https://issues.apache.org/jira/browse/KAFKA-8824
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.3.0
Reporter: Ferran altimiras
 Attachments: Test.java

Maybe this is not a bug, but it looks like something is wrong. This didn't 
happen in kafka streams 2.2.

 

Applying an aggregate() with suppress on kafka 2.3 sends nulls into Serializer 
if delayed msgs are received.

Not sure if some data is lost or not(yet). But IMHO getting a null to serialize 
the "accumulator" object is suspicious that something is wrong.

 

Attached java code to demonstrate it.

With kafka 2.3 -> LongSerde prints NULL, not in kafka 2.2

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

[Mickael Maison]

+1 (non binding)
Thanks Jason

On Wed, Aug 21, 2019 at 8:15 AM David Jacot  wrote:
>
> +1 (non-binding)
>
> Thanks for the KIP!
>
> Best,
> David
>
> On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson  wrote:
>
> > Hi All,
> >
> > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455 to fix
> > a long-known shortcoming of URP reporting and to improve reassignment
> > monitoring:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> > .
> >
> > Note that I have added one new metric following the discussion. It seemed
> > useful to have a lag metric for reassigning partitions.
> >
> > Thanks,
> > Jason
> >

[jira] [Created] (KAFKA-8823) Retention is partially broken on recreated topic

[Gregory Koshelev (Jira)]

Gregory Koshelev created KAFKA-8823:
---

 Summary: Retention is partially broken on recreated topic
 Key: KAFKA-8823
 URL: https://issues.apache.org/jira/browse/KAFKA-8823
 Project: Kafka
  Issue Type: Bug
  Components: log cleaner
Affects Versions: 2.2.0
Reporter: Gregory Koshelev


I've recreated topic with 48 partitions across 6 brokers with replication 
factor 3 with following config:
{code}
retention.ms=8640
retention.bytes=137438953472
{code}

Log cleaner have cleaned old segments in most partitions after 1 day as 
expected. But some partitions remained untouched (those partitions still have 
segments with base offset 0).

There are no errors in the server.log or log-cleaner.log for a week after topic 
creation. Also, some partitions remains untouched and grows over 1TB. There no 
messages for broken partitions in the server.log like:
{code}
[2019-08-14 23:55:35,465] INFO [Log partition=traces_prod-25, 
dir=/storage6/kafka/data] Found deletable segments with base offsets 
[355575520] due to retention size in bytes 137438953472 breach (kafka.log.Log)
[2019-08-14 23:55:35,465] INFO [Log partition=traces_prod-25, 
dir=/storage6/kafka/data] Scheduling log segment [baseOffset 355575520, size 
1073739552] for deletion. (kafka.log.Log)
[2019-08-14 23:55:35,465] INFO [Log partition=traces_prod-25, 
dir=/storage6/kafka/data] Incrementing log start offset to 358210127 
(kafka.log.Log)
{code}

So, finally, what I have tried:
# Set up another retention.ms (no success).
# Set up retention.bytes (no success).
# Restart the broker (success).

It's important to note that the topic had existed before: it was deleted and 
recreated again. I see some errors in the server.log about topic deletion:
{code}
[2019-08-13 18:31:24,402] INFO Deleted log 
/storage7/kafka/data/traces_prod-16.8395523b3a25454e8a1a7dd35c5f15d3-delete/.log.
 (kafka.log.LogSegment)
[2019-08-13 18:31:24,406] ERROR Error unmapping index 
/storage7/kafka/data/traces_prod-16.8395523b3a25454e8a1a7dd35c5f15d3-delete/.index
 (kafka.log.OffsetIndex)
java.lang.NullPointerException
at 
org.apache.kafka.common.utils.MappedByteBuffers.unmap(MappedByteBuffers.java:73)
at kafka.log.AbstractIndex.forceUnmap(AbstractIndex.scala:321)
at kafka.log.AbstractIndex.safeForceUnmap(AbstractIndex.scala:311)
at 
kafka.log.AbstractIndex.$anonfun$closeHandler$1(AbstractIndex.scala:260)
at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.log.AbstractIndex.closeHandler(AbstractIndex.scala:260)
at kafka.log.AbstractIndex.deleteIfExists(AbstractIndex.scala:229)
at kafka.log.LogSegment.$anonfun$deleteIfExists$6(LogSegment.scala:597)
at kafka.log.LogSegment.delete$1(LogSegment.scala:585)
at kafka.log.LogSegment.$anonfun$deleteIfExists$5(LogSegment.scala:597)
at kafka.utils.CoreUtils$.$anonfun$tryAll$1(CoreUtils.scala:115)
at kafka.utils.CoreUtils$.$anonfun$tryAll$1$adapted(CoreUtils.scala:114)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.utils.CoreUtils$.tryAll(CoreUtils.scala:114)
at kafka.log.LogSegment.deleteIfExists(LogSegment.scala:599)
at kafka.log.Log.$anonfun$delete$3(Log.scala:1762)
at kafka.log.Log.$anonfun$delete$3$adapted(Log.scala:1762)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at kafka.log.Log.$anonfun$delete$2(Log.scala:1762)
at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.log.Log.maybeHandleIOException(Log.scala:2013)
at kafka.log.Log.delete(Log.scala:1759)
at kafka.log.LogManager.deleteLogs(LogManager.scala:761)
at kafka.log.LogManager.$anonfun$deleteLogs$6(LogManager.scala:775)
at 
kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec

Re: ACL for group creation?

[David Jacot]

Hello,

It would be better to ask such question on the user mailing list.

The reason is that the group is created automatically when a consumer
joins it. It is not created explicitly so it can be restricted.

In your case, you could setup a ACL to authorize the application to only
use the group you have defined. It would prevent the application from
creating new groups. (READ Acl on Group resource with a specific name).

Best,
David

On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare 
wrote:

> Hi All
>
> I am looking through the Confluent docs and core Kafka docs and don't see
> an ACL for group creation:
> https://docs.confluent.io/current/kafka/authorization.html#acl-format
> and
> https://kafka.apache.org/documentation/#security_authz
>
> My scenario is simple: We use the consumer group as the means of
> identifying a single application, including tooling for managing
> application resets, offset management, lag monitoring, etc. We often have
> situations where someone resets their consumer group by appending an
> incremented integer ("cg" to "cg1"), but it throws the rest of the
> monitoring and management tooling out of whack.
>
> Is there a reason why we do not have ACL-based CREATE restrictions to a
> particular consumer group? I am willing to do the work to implement this
> and test it out, but I wanted to validate that there isn't a reason I am
> missing.
>
> Thanks
> Adam
>

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

[David Jacot]

+1 (non-binding)

Thanks for the KIP!

Best,
David

On Tue, Aug 20, 2019 at 7:55 PM Jason Gustafson  wrote:

> Hi All,
>
> I'd like to start a vote on KIP-352, which is a follow-up to KIP-455 to fix
> a long-known shortcoming of URP reporting and to improve reassignment
> monitoring:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-352%3A+Distinguish+URPs+caused+by+reassignment
> .
>
> Note that I have added one new metric following the discussion. It seemed
> useful to have a lag metric for reassigning partitions.
>
> Thanks,
> Jason
>