Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Levani Kokhreidze]

Thank you all!

The vote has been open for ~8 days. KIP has three binding votes (Bill, 
Guozhang, Matthias) and one non-binding (Sophie) so the KIP vote passes!
I’ll mark KIP as accepted and start working on it as soon as possible!

Regards,
Levani

> On Aug 1, 2019, at 2:37 AM, Matthias J. Sax  wrote:
> 
> +1 (binding)
> 
> On 7/31/19 8:36 AM, Guozhang Wang wrote:
>> Thanks for the update! +1 (binding).
>> 
>> On Tue, Jul 30, 2019 at 11:42 PM Levani Kokhreidze 
>> wrote:
>> 
>>> Hello Guozhang,
>>> 
>>> Thanks for the feedback. That’s an interesting point. To be honest, I
>>> totally missed it. I wasn’t aware that there’s `groupBy` possibility on
>>> KTable.
>>> I don’t see any reasons why not to add same functionality to KTable
>>> interface.
>>> 
>>> I’ve updated the KIP:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
>>> <
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
 
>>> Please let me know if you have any other questions and/or concerns.
>>> 
>>> Regards,
>>> Levani
>>> 
 On Jul 31, 2019, at 1:24 AM, Guozhang Wang  wrote:
 
 Hello Levani,
 
 Thanks for the KIP! Just got a quick question here about the scope: why
>>> do
 we only want this for `KStream`, not `KTable#groupBy` for example?
 
 
 Guozhang
 
 
 On Tue, Jul 30, 2019 at 1:27 PM Bill Bejeck  wrote:
 
> Thanks for the KIP Levani.
> 
> +1 (binding)
> 
> -Bill
> 
> On Tue, Jul 30, 2019 at 3:37 PM Levani Kokhreidze <
>>> levani.co...@gmail.com>
> wrote:
> 
>> Hello,
>> 
>> Still waiting for feedback on this KIP.
>> Please let me know if you have any concerns and/or questions.
>> 
>> Regards,
>> Levani
>> 
>> 
>>> On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman >>> 
>> wrote:
>>> 
>>> Looks good! Thanks Levani,
>>> 
>>> +1 (non-binding)
>>> 
>>> Sophie
>>> 
>>> On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze <
>> levani.co...@gmail.com>
>>> wrote:
>>> 
 Hello,
 
 I’d like to initialize voting on KIP-221:
 
>> 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
 <
 
>> 
> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> 
 If there’re any more concerns about the KIP, happy to discuss
>>> further.
 
 Regards,
 Levani
>> 
>> 
> 
 
 
 --
 -- Guozhang
>>> 
>>> 
>> 
> 

Build failed in Jenkins: kafka-trunk-jdk8 #3822

[Apache Jenkins Server]

See 


Changes:

[wangguoz]  KAFKA-8179: PartitionAssignorAdapter (#7110)

[cmccabe] MINOR: Add RandomComponentPayloadGenerator and update Trogdor

[bbejeck] KAFKA-8731: InMemorySessionStore throws NullPointerException on 
startup

--
[...truncated 2.57 MB...]

org.apache.kafka.connect.data.SchemaBuilderTest > testStringBuilder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testStringBuilder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testDefaultFieldsDifferentValueOverwriting STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testDefaultFieldsDifferentValueOverwriting PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testMapValueSchemaNull STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testMapValueSchemaNull PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testBytesBuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testBytesBuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testNonStructCantHaveFields 
STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testNonStructCantHaveFields 
PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testDoubleBuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testDoubleBuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testFloatBuilder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testFloatBuilder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testFieldNameNull STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testFieldNameNull PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testBooleanBuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testBooleanBuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt16Builder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt16Builder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testInt16BuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testInt16BuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testBytesBuilder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testBytesBuilder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testStructBuilder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testStructBuilder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt64Builder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt64Builder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testDefaultFieldsSameValueOverwriting STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testDefaultFieldsSameValueOverwriting PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testMapBuilder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testMapBuilder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testParameters STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testParameters PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testArrayBuilder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testArrayBuilder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testArrayBuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testArrayBuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testDuplicateFields STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testDuplicateFields PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testStringBuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testStringBuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testEmptyStruct STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testEmptyStruct PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testMapBuilderInvalidDefault 
STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testMapBuilderInvalidDefault 
PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testInt64BuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testInt64BuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt8Builder STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt8Builder PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testFieldSchemaNull STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > testFieldSchemaNull PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testInt32BuilderInvalidDefault STARTED

org.apache.kafka.connect.data.SchemaBuilderTest > 
testInt32BuilderInvalidDefault PASSED

org.apache.kafka.connect.data.SchemaBuilderTest > testInt8BuilderInvalidDefault 
STARTED

org.apache.kafka.connect.data.Sch

Re: Request to be added as a contributor

[Guozhang Wang]

Hello,

I've added you as a contributor to Kafka.

Cheers,
Guozhang


On Wed, Jul 31, 2019 at 5:31 PM Tharindu Amila <
tharindu.am...@instaclustr.com> wrote:

> Hi Team,
>
> Request you to add me to the contributor list. Jira username : tharindu
>
> Thanks and Regards,
>
>
> *Tharindu Amila**Software Engineer*
>
>
>    
> 
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
> Instaclustr values your privacy. Our privacy policy can be found at
> https://www.instaclustr.com/company/policies/privacy-policy
>


-- 
-- Guozhang

Jenkins build is back to normal : kafka-2.3-jdk8 #79

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk11 #724

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Small cleanups in TopicCommand describe handling (#7136)

[junrao] Update KafkaConfig.scala (#7113)

[wangguoz]  KAFKA-8179: PartitionAssignorAdapter (#7110)

[cmccabe] MINOR: Add RandomComponentPayloadGenerator and update Trogdor

[bbejeck] KAFKA-8731: InMemorySessionStore throws NullPointerException on 
startup

--
[...truncated 2.58 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = true] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = true] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopic[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = true] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = true] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabl

Build failed in Jenkins: kafka-trunk-jdk8 #3821

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Tolerate limited data loss for upgrade tests with old message

[github] MINOR: Small cleanups in TopicCommand describe handling (#7136)

[junrao] Update KafkaConfig.scala (#7113)

--
[...truncated 9.03 MB...]

org.apache.kafka.connect.runtime.ConnectorConfigTest > unconfiguredTransform 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > 
multipleTransformsOneDangling STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > 
multipleTransformsOneDangling PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > misconfiguredTransform 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > misconfiguredTransform 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > noTransforms STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > noTransforms PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > danglingTransformAlias 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > danglingTransformAlias 
PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > emptyConnectorName 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > emptyConnectorName PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > multipleTransforms 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > multipleTransforms PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > singleTransform PASSED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
STARTED

org.apache.kafka.connect.runtime.ConnectorConfigTest > wrongTransformationType 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testGettingGroupMultipleTimes STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testGettingGroupMultipleTimes PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testCreatingTags STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testCreatingTags PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testMetricGroupIdWithoutTags STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testMetricGroupIdWithoutTags PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testCreatingTagsWithOddNumberOfTags STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testCreatingTagsWithOddNumberOfTags PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testGettingGroupWithOddNumberOfTags STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > 
testGettingGroupWithOddNumberOfTags PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testMetricGroupIdIdentity 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testMetricGroupIdIdentity 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithoutClose 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithoutClose 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testGettingGroupWithTags 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testGettingGroupWithTags 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithClose 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testRecreateWithClose 
PASSED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testKafkaMetricsNotNull 
STARTED

org.apache.kafka.connect.runtime.ConnectMetricsTest > testKafkaMetricsNotNull 
PASSED

org.apache.kafka.connect.runtime.isolation.DelegatingClassLoaderTest > 
testWhiteListedManifestResources STARTED

org.apache.kafka.connect.runtime.isolation.DelegatingClassLoaderTest > 
testWhiteListedManifestResources PASSED

org.apache.kafka.connect.runtime.isolation.DelegatingClassLoaderTest > 
testOtherResources STARTED

org.apache.kafka.connect.runtime.isolation.DelegatingClassLoaderTest > 
testOtherResources PASSED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescWithNullVersion STARTED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescWithNullVersion PASSED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescComparison STARTED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescComparison PASSED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testRegularPluginDesc STARTED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testRegularPluginDesc PASSED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescEquality STARTED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescEquality PASSED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescWithSystemClassLoader STARTED

org.apache.kafka.connect.runtime.isolation.PluginDescTest > 
testPluginDescWithSystemClassL

Request to be added as a contributor

[Tharindu Amila]

Hi Team,

Request you to add me to the contributor list. Jira username : tharindu

Thanks and Regards,


*Tharindu Amila**Software Engineer*


   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Instaclustr values your privacy. Our privacy policy can be found at
https://www.instaclustr.com/company/policies/privacy-policy

Dynamic configuration of interbroker SSL certificates

[Michael Carter]

Hi all,

I'm having an issue with dynamic configuration of interbroker SSL certificates 
(in Kafka 2.3.0) that I'm hoping someone can give me insight on. I've 
previously posted something similar on the Users email list, but I think it 
will need some help from developers experienced with how the networking code in 
Kafka works.

I'm trying to use SSL two-way authentication for inter broker communication, 
with short lived SSL certificates, rotatating them frequently without needing 
to do a broker restart. So, on each broker in my cluster, I periodically 
generate a new certificate keystore file, and set the 
"listener.name.interbroker.ssl.keystore.location" broker config property 
property. (I'm using inter.broker.listener.name=INTERBROKER)

Setting this property works fine, and everything appears ok. And manually 
connecting to the inter broker listener shows it's correctly serving the new 
certificate. But if I ever restart a broker after the original certificate has 
expired (The one the broker started up with, which is no longer configured 
anywhere), then communication failures between brokers start to toccur. My logs 
fill up with messages like this:

[2019-07-22 03:57:43,605] INFO [SocketServer brokerId=1] Failed authentication 
with 10.224.70.3 (SSL handshake failed) 
(org.apache.kafka.common.network.Selector)

A little bit of extra logging injected into the code tells me that the failures 
are caused by the out of date SSL certificates being used. So it seems there 
are some network components inside Kafka still stuck on the old settings.
This sounds similar to the behaviour described in KAFKA-8336 
(https://issues.apache.org/jira/browse/KAFKA-8336), but this is marked as fixed 
in 2.3.0.

I've confirmed that all the SslChannelBuilders and SslFactories appear to be 
being reconfigured correctly when the dynamic setting is set. I've tried 
closing all existing KafkaChannels on a reconfiguration event, in order to 
force them to re-open with the new certificates, but the problem persists.

Does anyone have any idea what other components may be hanging around, using 
the old certificates?

Thanks,
Michael

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Matthias J. Sax]

+1 (binding)

On 7/31/19 8:36 AM, Guozhang Wang wrote:
> Thanks for the update! +1 (binding).
> 
> On Tue, Jul 30, 2019 at 11:42 PM Levani Kokhreidze 
> wrote:
> 
>> Hello Guozhang,
>>
>> Thanks for the feedback. That’s an interesting point. To be honest, I
>> totally missed it. I wasn’t aware that there’s `groupBy` possibility on
>> KTable.
>> I don’t see any reasons why not to add same functionality to KTable
>> interface.
>>
>> I’ve updated the KIP:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
>>>
>> Please let me know if you have any other questions and/or concerns.
>>
>> Regards,
>> Levani
>>
>>> On Jul 31, 2019, at 1:24 AM, Guozhang Wang  wrote:
>>>
>>> Hello Levani,
>>>
>>> Thanks for the KIP! Just got a quick question here about the scope: why
>> do
>>> we only want this for `KStream`, not `KTable#groupBy` for example?
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Tue, Jul 30, 2019 at 1:27 PM Bill Bejeck  wrote:
>>>
 Thanks for the KIP Levani.

 +1 (binding)

 -Bill

 On Tue, Jul 30, 2019 at 3:37 PM Levani Kokhreidze <
>> levani.co...@gmail.com>
 wrote:

> Hello,
>
> Still waiting for feedback on this KIP.
> Please let me know if you have any concerns and/or questions.
>
> Regards,
> Levani
>
>
>> On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman >>
> wrote:
>>
>> Looks good! Thanks Levani,
>>
>> +1 (non-binding)
>>
>> Sophie
>>
>> On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze <
> levani.co...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I’d like to initialize voting on KIP-221:
>>>
>

>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
>>> <
>>>
>

>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint

>>> If there’re any more concerns about the KIP, happy to discuss
>> further.
>>>
>>> Regards,
>>> Levani
>
>

>>>
>>>
>>> --
>>> -- Guozhang
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Jenkins build is back to normal : kafka-trunk-jdk11 #723

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-8704) Add PartitionAssignor adapter for backwards compatibility

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8704.
--
   Resolution: Fixed
Fix Version/s: 2.4.0

> Add PartitionAssignor adapter for backwards compatibility
> -
>
> Key: KAFKA-8704
> URL: https://issues.apache.org/jira/browse/KAFKA-8704
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Reporter: Sophie Blee-Goldman
>Assignee: Sophie Blee-Goldman
>Priority: Major
> Fix For: 2.4.0
>
>
> As part of KIP-429, we are deprecating the old 
> consumer.internal.PartitionAssignor in favor of a [new 
> consumer.PartitionAssignor|https://issues.apache.org/jira/browse/KAFKA-8703] 
> interface  that is part of the public API.
>  
> Although the old PartitionAssignor was technically part of the internal 
> package, some users may have implemented it and this change will break source 
> compatibility for them as they would need to modify their class to implement 
> the new interface. The number of users affected may be small, but nonetheless 
> we would like to add an adapter to maintain compatibility for these users.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Colin McCabe]

Hi Mickael,

I think you bring up a good point.  It would be better if we didn't ever have 
to set up client-side configuration for this feature, and KIP-464 would let us 
skip this entirely.

On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote:
> Hi Mickael,
> I agree that KIP-464 works on newer brokers, but I was a bit worried how
> things would play out on older brokers that* do not *have KIP 464 included.
> Is it enough to throw an error in this case when producer configs are not
> specified?

I think the right thing to do would be to log an error message in the client.  
We will need to have that capability in any case, to cover scenarios like the 
client trying to auto-create a topic that they don't have permission to create. 
 Or a client trying to create a topic on a broker so old that 
CreateTopicsRequest is not supported.

The big downside to relying on KIP-464 is that it is a very recent feature -- 
so recent that it hasn't even made its way to any official Apache release.  
It's scheduled for the upcoming 2.4 release in a few months.

So if you view this KIP as a step towards removing broker-side auto-create, you 
might want to support older brokers just to accelerate adoption, and hasten the 
day when we can finally flip broker-side auto-create to off (or even remove it 
entirely).

I have to agree, though, that having client-side configurations for number of 
partitions and replication factor is messy.  Maybe it would be worth it to 
restrict support to post-KIP-464 brokers, if we could avoid creating more 
configs.

best,
Colin


> On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison 
> wrote:
> 
> > Hi Justine,
> >
> > We can rely on KIP-464 which allows to omit the partition count or
> > replication factor when creating a topic. In that case, the broker
> > defaults are used.
> >
> > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan 
> > wrote:
> > >
> > > Michael,
> > > That makes sense to me!
> > > To clarify, in the current state of the KIP, the producer does not rely
> > on
> > > the broker to autocreate--if the broker's config is disabled, then the
> > > producer can autocreate on its own with a create topic request (the same
> > > type of request the admin client uses).
> > > However, if both configs are enabled, the broker will autocreate through
> > a
> > > metadata request before the producer gets a chance.
> > > Of course, the way to avoid this, is to do as you suggested, and set the
> > > "allow_auto_topic_creation" field to false.
> > >
> > > I think the only thing we need to be careful with in this setup is
> > without
> > > KIP 464, we can not use broker defaults for this topic. A user needs to
> > > specify the number of partition and replication factor in the config.
> > > An alternative to this is to have coded defaults for when these configs
> > are
> > > unspecified, but it is not immediately apparent what these defaults
> > should
> > > be.
> > >
> > > Thanks again for reading my KIP,
> > > Justine
> > >
> > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison  > >
> > > wrote:
> > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the response!
> > > > In my opinion, it would be better if the producer did not rely at all
> > > > on the broker auto create feature as this is what we're aiming to
> > > > deprecate. When requesting metadata we can set the
> > > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > > creation. Then if the topic is not existing, send a
> > > > CreateTopicRequest.
> > > >
> > > > What do you think?
> > > >
> > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan 
> > > > wrote:
> > > > >
> > > > > Currently the way it is implemented, the broker auto-creation
> > > > configuration
> > > > > takes precedence. The producer will not use the CreateTopics request.
> > > > > (Technically it can--but the topic will already be created through
> > the
> > > > > broker, so it will never try to create the topic.)
> > > > > It is possible to change this however, and I'd be happy to discuss
> > the
> > > > > benefits of this alternative.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the KIP!
> > > > > >
> > > > > > In case auto creation is enabled on both the client and server,
> > will
> > > > > > the producer still use the AdminClient (CreateTopics request) to
> > > > > > create topics? and not rely on the broker auto create.
> > > > > > I'm guessing the answer is yes but can you make it explicit.
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> > jols...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > > Just a friendly reminder to take a look at this KIP if you have
> > the
> > > > time.
> > > > > > >
> > > > > > > I was thinking about broker vs. client default precedenc

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

[Colin McCabe]

Hi Jun,

Thanks for taking another look at this.

On Wed, Jul 31, 2019, at 09:22, Jun Rao wrote:
> Hi, Stan,
> 
> Thanks for the explanation.
> 
> 10. If those new fields in LeaderAndIsr are only needed for future work,
> perhaps they should be added when we do the future work instead of now?

I think this ties in with one of the big goals of this KIP, making it possible 
to distinguish reassigning replicas from normal replicas.  This is the key to 
follow-on work like being able to ensure that partitions with a reassignment 
don't get falsely flagged as under-replicated in the metrics, or implementing 
reassignment quotas that don't accidentally affect normal replication traffic 
when a replica falls out of the ISR.

For these follow-on improvements, we need to have that information in 
LeaderAndIsrRequest.  We could add the information in a follow-on KIP, of 
course, but then all the improvements are blocked on that follow-on KIP.  That 
would slow things down for all of the downstream KIPs that are blocked on this.

Also, to keep things consistent, I think it would be best if the format of the 
data in the LeaderAndIsrRequest matched the format of the data in ZooKeeper.  
Since we're deciding on the ZK format in this KIP, I think it makes sense to 
also decide on the format in the LeaderAndIsrRequest.

> > > Should we include those two fields in UpdateMetadata and potentially
> > > Metadata requests too?

We had some discussion earlier about how metadata responses to clients are 
getting too large, in part because they include a lot of information that most 
clients don't need (such as the ISR).  I think reassignment information 
definitely falls in the category of something a client doesn't need to know, so 
we shouldn't include it.

A program like CruiseControl, or the command-line reassignment program, just 
wants to get the most up-to-date information about the state of reassigning 
partitions.  The MetadataRequest API wouldn't deliver that, because there are 
inherently delays in how we propagate metadata to brokers.  That's why the 
ListPartitionReassignments API is a better choice for those programs.  So I 
think if we added this information to the MetadataResponse, nobody would 
actually use it, and it would just use up more bandwidth.

Of course, we can always revisit this later if we find a scenario where a 
producer or consumer would actually care about this.  But I think we should 
default to not adding stuff to the metadata response if we don't have a good 
use case in mind.

> > > 11. "If a new reassignment is issued during an on-going one, we cancel the
> > > current one by emptying out both AR and RR, constructing them from (the
> > > updated from the last-reassignment) R and TR, and starting anew." In this
> > > case, it seems that the controller needs to issue a StopReplica request to
> > > remove those unneeded replicas.

Good catch.  Yes, we should document this in the  KIP.

> > > 12. "Essentially, once a cancellation is called we subtract AR from R,
> > > empty out both AR and RR, and send LeaderAndIsr requests to cancel the
> > > replica movements that have not yet completed." Similar to the above, it
> > > seems the controller needs to issue a StopReplica request to remove those
> > > unneeded replicas.

Right.  Let's add this.

> > > 13. Since we changed the format of the topics/[topic] zNode, should we 
> > > bump
> > > up the version number in the json value?

The change to the zNode is backwards compatible, though.  Older brokers will 
continue to work, but just ignore the new fields.  If we bump that version 
number, then downgrades will require hand-editing zookeeper.  (Of course 
downgrade isn't officially supported, but it would be nice not to break it if 
we don't need to...)  Changing the version number would also create problems 
during a rolling upgrade.

best,
Colin

> > >
> > > Jun
> > >
> > > On Mon, Jul 22, 2019 at 8:38 AM Colin McCabe  wrote:
> > >
> > > > Hi all,
> > > >
> > > > With three non-binding +1 votes from Viktor Somogyi-Vass, Robert
> > Barrett,
> > > > and George Li, and 3 binding +1 votes from Gwen Shapira, Jason
> > Gustafson,
> > > > and myself, the vote passes.  Thanks, everyone!
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Fri, Jul 19, 2019, at 08:55, Robert Barrett wrote:
> > > > > +1 (non-binding). Thanks for the KIP!
> > > > >
> > > > > On Thu, Jul 18, 2019 at 5:59 PM George Li  > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > >  +1 (non-binding)
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks for addressing the comments.
> > > > > > George
> > > > > >
> > > > > > On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen Shapira <
> > > > > > g...@confluent.io> wrote:
> > > > > >
> > > > > >  Renewing my +1, thank you Colin and Stan for working through all
> > the
> > > > > > questions, edge cases, requests and alternatives. We ended up with
> > a
> > > > > > great protocol.
> > > > > >
> > > > > > On Thu, Jul 18, 2019 at 4:54

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

[Jun Rao]

Hi, Stan,

Thanks for the explanation.

10. If those new fields in LeaderAndIsr are only needed for future work,
perhaps they should be added when we do the future work instead of now?

Jun


On Wed, Jul 31, 2019 at 2:30 AM Stanislav Kozlovski 
wrote:

> Hey Jun,
>
> I think I can answer some of your questions on behalf of Colin -- he can
> confirm if I'm correct
>
> > 10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to
> LeaderAndIsr request. Could you explain how these 2 fields will be used?
> Sorry for not explaining this in the KIP - those fields won't be used by
> the non-controller brokers yet. Our plans for them are outlined in the
> Future Work section of the KIP - namely "Reassignment Quotas that only
> throttle reassignment traffic" and "Add reassignment metrics".
>
> > Should we include those two fields in UpdateMetadata and potentially
> Metadata requests too?
> I recall this was discussed in the beginning by Colin and Jason, so I'll
> let Colin answer that question.
>
> 11 & 12. Correct, we need to send StopReplica requests. The implementation
> does this (
>
> https://github.com/apache/kafka/pull/7128/files#diff-ed90e8ecc5439a5ede5e362255d11be1R651
> )
> -- I'll update the KIP to mention it as well.
> I tried to document the algorithm here
>
> https://github.com/apache/kafka/pull/7128/files#diff-ed90e8ecc5439a5ede5e362255d11be1R521
> .
>
> 13. I think so. (
> https://github.com/apache/kafka/pull/7128#discussion_r308866206). I'll
> reflect this in the KIP
>
> Here is the updated KIP diff -
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=112820260&selectedPageVersions=36&selectedPageVersions=35
>
> Thanks,
> Stanislav
>
> On Tue, Jul 30, 2019 at 10:18 PM Jun Rao  wrote:
>
> > Hi, Colin,
> >
> > Thanks for the KIP. Sorry for the late reply. LGTM overall. A few
> detailed
> > comments below.
> >
> > 10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to
> > LeaderAndIsr request. Could you explain how these 2 fields will be used?
> > Should we include those two fields in UpdateMetadata and potentially
> > Metadata requests too?
> >
> > 11. "If a new reassignment is issued during an on-going one, we cancel
> the
> > current one by emptying out both AR and RR, constructing them from (the
> > updated from the last-reassignment) R and TR, and starting anew." In this
> > case, it seems that the controller needs to issue a StopReplica request
> to
> > remove those unneeded replicas.
> >
> > 12. "Essentially, once a cancellation is called we subtract AR from R,
> > empty out both AR and RR, and send LeaderAndIsr requests to cancel the
> > replica movements that have not yet completed." Similar to the above, it
> > seems the controller needs to issue a StopReplica request to remove those
> > unneeded replicas.
> >
> > 13. Since we changed the format of the topics/[topic] zNode, should we
> bump
> > up the version number in the json value?
> >
> > Jun
> >
> > On Mon, Jul 22, 2019 at 8:38 AM Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > With three non-binding +1 votes from Viktor Somogyi-Vass, Robert
> Barrett,
> > > and George Li, and 3 binding +1 votes from Gwen Shapira, Jason
> Gustafson,
> > > and myself, the vote passes.  Thanks, everyone!
> > >
> > > best,
> > > Colin
> > >
> > > On Fri, Jul 19, 2019, at 08:55, Robert Barrett wrote:
> > > > +1 (non-binding). Thanks for the KIP!
> > > >
> > > > On Thu, Jul 18, 2019 at 5:59 PM George Li  > > .invalid>
> > > > wrote:
> > > >
> > > > >  +1 (non-binding)
> > > > >
> > > > >
> > > > >
> > > > > Thanks for addressing the comments.
> > > > > George
> > > > >
> > > > > On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen Shapira <
> > > > > g...@confluent.io> wrote:
> > > > >
> > > > >  Renewing my +1, thank you Colin and Stan for working through all
> the
> > > > > questions, edge cases, requests and alternatives. We ended up with
> a
> > > > > great protocol.
> > > > >
> > > > > On Thu, Jul 18, 2019 at 4:54 PM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > +1 Thanks for the KIP. Really looking forward to this!
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Jul 17, 2019 at 1:41 PM Colin McCabe  >
> > > wrote:
> > > > > >
> > > > > > > Thanks, Stanislav.  Let's restart the vote to reflect the fact
> > that
> > > > > we've
> > > > > > > made significant changes.  The new vote will go for 3 days as
> > > usual.
> > > > > > >
> > > > > > > I'll start with my +1 (binding).
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 17, 2019, at 08:56, Stanislav Kozlovski wrote:
> > > > > > > > Hey everybody,
> > > > > > > >
> > > > > > > > We have further iterated on the KIP in the accompanying
> > > discussion
> > > > > thread
> > > > > > > > and I'd like to propose we resume the vote.
> > > > > > > >
> > > > > > > > Some notable changes:
> > > > > > > > - we will store reassignment informati

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Justine Olshan]

Hi Mickael,
I agree that KIP-464 works on newer brokers, but I was a bit worried how
things would play out on older brokers that* do not *have KIP 464 included.
Is it enough to throw an error in this case when producer configs are not
specified?

Thanks,
Justine

On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison 
wrote:

> Hi Justine,
>
> We can rely on KIP-464 which allows to omit the partition count or
> replication factor when creating a topic. In that case, the broker
> defaults are used.
>
> On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan 
> wrote:
> >
> > Michael,
> > That makes sense to me!
> > To clarify, in the current state of the KIP, the producer does not rely
> on
> > the broker to autocreate--if the broker's config is disabled, then the
> > producer can autocreate on its own with a create topic request (the same
> > type of request the admin client uses).
> > However, if both configs are enabled, the broker will autocreate through
> a
> > metadata request before the producer gets a chance.
> > Of course, the way to avoid this, is to do as you suggested, and set the
> > "allow_auto_topic_creation" field to false.
> >
> > I think the only thing we need to be careful with in this setup is
> without
> > KIP 464, we can not use broker defaults for this topic. A user needs to
> > specify the number of partition and replication factor in the config.
> > An alternative to this is to have coded defaults for when these configs
> are
> > unspecified, but it is not immediately apparent what these defaults
> should
> > be.
> >
> > Thanks again for reading my KIP,
> > Justine
> >
> > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison  >
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for the response!
> > > In my opinion, it would be better if the producer did not rely at all
> > > on the broker auto create feature as this is what we're aiming to
> > > deprecate. When requesting metadata we can set the
> > > "allow_auto_topic_creation" field to false to avoid the broker auto
> > > creation. Then if the topic is not existing, send a
> > > CreateTopicRequest.
> > >
> > > What do you think?
> > >
> > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan 
> > > wrote:
> > > >
> > > > Currently the way it is implemented, the broker auto-creation
> > > configuration
> > > > takes precedence. The producer will not use the CreateTopics request.
> > > > (Technically it can--but the topic will already be created through
> the
> > > > broker, so it will never try to create the topic.)
> > > > It is possible to change this however, and I'd be happy to discuss
> the
> > > > benefits of this alternative.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > In case auto creation is enabled on both the client and server,
> will
> > > > > the producer still use the AdminClient (CreateTopics request) to
> > > > > create topics? and not rely on the broker auto create.
> > > > > I'm guessing the answer is yes but can you make it explicit.
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <
> jols...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > > Just a friendly reminder to take a look at this KIP if you have
> the
> > > time.
> > > > > >
> > > > > > I was thinking about broker vs. client default precedence, and I
> > > think it
> > > > > > makes sense to keep the broker as the default used when both
> > > client-side
> > > > > > and broker-side defaults are configured. The idea is that there
> > > would be
> > > > > > pretty clear documentation, and that many systems with
> configurations
> > > > > that
> > > > > > the client could not change would likely have the auto-create
> default
> > > > > off.
> > > > > > (In cloud for example).
> > > > > >
> > > > > > It also seems like in most cases, the consumer config
> > > > > > 'allow.auto.create.topics' was created to actually prevent the
> > > creation
> > > > > of
> > > > > > topics, so the loss of creation functionality will not be a big
> > > problem.
> > > > > >
> > > > > >  I'm happy to discuss any other compatibility problems or
> components
> > > of
> > > > > > this KIP.
> > > > > >
> > > > > > Thank you,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <
> jols...@confluent.io
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I was looking at this KIP again, and there is a decision I made
> > > that I
> > > > > > > think is worth discussing.
> > > > > > >
> > > > > > > In the case where both the broker and producer's
> > > > > > > 'auto.create.topics.enable' are set to true, we have to choose
> > > either
> > > > > the
> > > > > > > broker configs or the producer configs for the replication
> > > > > > > factor/partitions.
> > > > > > >
> 

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Mickael Maison]

Hi Justine,

We can rely on KIP-464 which allows to omit the partition count or
replication factor when creating a topic. In that case, the broker
defaults are used.

On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan  wrote:
>
> Michael,
> That makes sense to me!
> To clarify, in the current state of the KIP, the producer does not rely on
> the broker to autocreate--if the broker's config is disabled, then the
> producer can autocreate on its own with a create topic request (the same
> type of request the admin client uses).
> However, if both configs are enabled, the broker will autocreate through a
> metadata request before the producer gets a chance.
> Of course, the way to avoid this, is to do as you suggested, and set the
> "allow_auto_topic_creation" field to false.
>
> I think the only thing we need to be careful with in this setup is without
> KIP 464, we can not use broker defaults for this topic. A user needs to
> specify the number of partition and replication factor in the config.
> An alternative to this is to have coded defaults for when these configs are
> unspecified, but it is not immediately apparent what these defaults should
> be.
>
> Thanks again for reading my KIP,
> Justine
>
> On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison 
> wrote:
>
> > Hi Justine,
> >
> > Thanks for the response!
> > In my opinion, it would be better if the producer did not rely at all
> > on the broker auto create feature as this is what we're aiming to
> > deprecate. When requesting metadata we can set the
> > "allow_auto_topic_creation" field to false to avoid the broker auto
> > creation. Then if the topic is not existing, send a
> > CreateTopicRequest.
> >
> > What do you think?
> >
> > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan 
> > wrote:
> > >
> > > Currently the way it is implemented, the broker auto-creation
> > configuration
> > > takes precedence. The producer will not use the CreateTopics request.
> > > (Technically it can--but the topic will already be created through the
> > > broker, so it will never try to create the topic.)
> > > It is possible to change this however, and I'd be happy to discuss the
> > > benefits of this alternative.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Justine,
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > In case auto creation is enabled on both the client and server, will
> > > > the producer still use the AdminClient (CreateTopics request) to
> > > > create topics? and not rely on the broker auto create.
> > > > I'm guessing the answer is yes but can you make it explicit.
> > > >
> > > > Thank you
> > > >
> > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan 
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > > Just a friendly reminder to take a look at this KIP if you have the
> > time.
> > > > >
> > > > > I was thinking about broker vs. client default precedence, and I
> > think it
> > > > > makes sense to keep the broker as the default used when both
> > client-side
> > > > > and broker-side defaults are configured. The idea is that there
> > would be
> > > > > pretty clear documentation, and that many systems with configurations
> > > > that
> > > > > the client could not change would likely have the auto-create default
> > > > off.
> > > > > (In cloud for example).
> > > > >
> > > > > It also seems like in most cases, the consumer config
> > > > > 'allow.auto.create.topics' was created to actually prevent the
> > creation
> > > > of
> > > > > topics, so the loss of creation functionality will not be a big
> > problem.
> > > > >
> > > > >  I'm happy to discuss any other compatibility problems or components
> > of
> > > > > this KIP.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan  > >
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I was looking at this KIP again, and there is a decision I made
> > that I
> > > > > > think is worth discussing.
> > > > > >
> > > > > > In the case where both the broker and producer's
> > > > > > 'auto.create.topics.enable' are set to true, we have to choose
> > either
> > > > the
> > > > > > broker configs or the producer configs for the replication
> > > > > > factor/partitions.
> > > > > >
> > > > > > Currently, the decision is to have the broker defaults take
> > precedence.
> > > > > > (It is easier to do this in the implementation.) It also makes some
> > > > sense
> > > > > > for this behavior to take precedence since this behavior already
> > > > occurs as
> > > > > > the default.
> > > > > >
> > > > > > However, I was wondering if it would be odd for those who can only
> > see
> > > > the
> > > > > > producer side to set configs for replication factor/partitions and
> > see
> > > > > > different results. Currently the documentation for the config
> > states
> > > > that
> > > > > > the config values are only used whe

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Justine Olshan]

Michael,
That makes sense to me!
To clarify, in the current state of the KIP, the producer does not rely on
the broker to autocreate--if the broker's config is disabled, then the
producer can autocreate on its own with a create topic request (the same
type of request the admin client uses).
However, if both configs are enabled, the broker will autocreate through a
metadata request before the producer gets a chance.
Of course, the way to avoid this, is to do as you suggested, and set the
"allow_auto_topic_creation" field to false.

I think the only thing we need to be careful with in this setup is without
KIP 464, we can not use broker defaults for this topic. A user needs to
specify the number of partition and replication factor in the config.
An alternative to this is to have coded defaults for when these configs are
unspecified, but it is not immediately apparent what these defaults should
be.

Thanks again for reading my KIP,
Justine

On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison 
wrote:

> Hi Justine,
>
> Thanks for the response!
> In my opinion, it would be better if the producer did not rely at all
> on the broker auto create feature as this is what we're aiming to
> deprecate. When requesting metadata we can set the
> "allow_auto_topic_creation" field to false to avoid the broker auto
> creation. Then if the topic is not existing, send a
> CreateTopicRequest.
>
> What do you think?
>
> On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan 
> wrote:
> >
> > Currently the way it is implemented, the broker auto-creation
> configuration
> > takes precedence. The producer will not use the CreateTopics request.
> > (Technically it can--but the topic will already be created through the
> > broker, so it will never try to create the topic.)
> > It is possible to change this however, and I'd be happy to discuss the
> > benefits of this alternative.
> >
> > Thank you,
> > Justine
> >
> > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for the KIP!
> > >
> > > In case auto creation is enabled on both the client and server, will
> > > the producer still use the AdminClient (CreateTopics request) to
> > > create topics? and not rely on the broker auto create.
> > > I'm guessing the answer is yes but can you make it explicit.
> > >
> > > Thank you
> > >
> > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan 
> > > wrote:
> > > >
> > > > Hi,
> > > > Just a friendly reminder to take a look at this KIP if you have the
> time.
> > > >
> > > > I was thinking about broker vs. client default precedence, and I
> think it
> > > > makes sense to keep the broker as the default used when both
> client-side
> > > > and broker-side defaults are configured. The idea is that there
> would be
> > > > pretty clear documentation, and that many systems with configurations
> > > that
> > > > the client could not change would likely have the auto-create default
> > > off.
> > > > (In cloud for example).
> > > >
> > > > It also seems like in most cases, the consumer config
> > > > 'allow.auto.create.topics' was created to actually prevent the
> creation
> > > of
> > > > topics, so the loss of creation functionality will not be a big
> problem.
> > > >
> > > >  I'm happy to discuss any other compatibility problems or components
> of
> > > > this KIP.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan  >
> > > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I was looking at this KIP again, and there is a decision I made
> that I
> > > > > think is worth discussing.
> > > > >
> > > > > In the case where both the broker and producer's
> > > > > 'auto.create.topics.enable' are set to true, we have to choose
> either
> > > the
> > > > > broker configs or the producer configs for the replication
> > > > > factor/partitions.
> > > > >
> > > > > Currently, the decision is to have the broker defaults take
> precedence.
> > > > > (It is easier to do this in the implementation.) It also makes some
> > > sense
> > > > > for this behavior to take precedence since this behavior already
> > > occurs as
> > > > > the default.
> > > > >
> > > > > However, I was wondering if it would be odd for those who can only
> see
> > > the
> > > > > producer side to set configs for replication factor/partitions and
> see
> > > > > different results. Currently the documentation for the config
> states
> > > that
> > > > > the config values are only used when the broker config is not
> enabled,
> > > but
> > > > > this might not always be clear to the user.  Changing the code to
> have
> > > the
> > > > > producer's configurations take precedence is possible, but I was
> > > wondering
> > > > > what everyone thought.
> > > > >
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <
> jols...@confluent.io>
> > > > > wrote:
> > > > >
> > > > >> Just a quick update--
> > > > >>
> > > 

Re: [VOTE] KIP-221: Enhance KStream with Connecting Topic Creation and Repartition Hint

[Guozhang Wang]

Thanks for the update! +1 (binding).

On Tue, Jul 30, 2019 at 11:42 PM Levani Kokhreidze 
wrote:

> Hello Guozhang,
>
> Thanks for the feedback. That’s an interesting point. To be honest, I
> totally missed it. I wasn’t aware that there’s `groupBy` possibility on
> KTable.
> I don’t see any reasons why not to add same functionality to KTable
> interface.
>
> I’ve updated the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint
> >
> Please let me know if you have any other questions and/or concerns.
>
> Regards,
> Levani
>
> > On Jul 31, 2019, at 1:24 AM, Guozhang Wang  wrote:
> >
> > Hello Levani,
> >
> > Thanks for the KIP! Just got a quick question here about the scope: why
> do
> > we only want this for `KStream`, not `KTable#groupBy` for example?
> >
> >
> > Guozhang
> >
> >
> > On Tue, Jul 30, 2019 at 1:27 PM Bill Bejeck  wrote:
> >
> >> Thanks for the KIP Levani.
> >>
> >> +1 (binding)
> >>
> >> -Bill
> >>
> >> On Tue, Jul 30, 2019 at 3:37 PM Levani Kokhreidze <
> levani.co...@gmail.com>
> >> wrote:
> >>
> >>> Hello,
> >>>
> >>> Still waiting for feedback on this KIP.
> >>> Please let me know if you have any concerns and/or questions.
> >>>
> >>> Regards,
> >>> Levani
> >>>
> >>>
>  On Jul 24, 2019, at 8:20 PM, Sophie Blee-Goldman  >
> >>> wrote:
> 
>  Looks good! Thanks Levani,
> 
>  +1 (non-binding)
> 
>  Sophie
> 
>  On Tue, Jul 23, 2019 at 2:16 PM Levani Kokhreidze <
> >>> levani.co...@gmail.com>
>  wrote:
> 
> > Hello,
> >
> > I’d like to initialize voting on KIP-221:
> >
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> > <
> >
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>
> > If there’re any more concerns about the KIP, happy to discuss
> further.
> >
> > Regards,
> > Levani
> >>>
> >>>
> >>
> >
> >
> > --
> > -- Guozhang
>
>

-- 
-- Guozhang

[jira] [Resolved] (KAFKA-8688) Upgrade system tests fail due to data loss with older message format

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-8688.
---
   Resolution: Fixed
 Reviewer: Ismael Juma
Fix Version/s: 2.4.0

> Upgrade system tests fail due to data loss with older message format
> 
>
> Key: KAFKA-8688
> URL: https://issues.apache.org/jira/browse/KAFKA-8688
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.4.0
>
>
> System test failure for TestUpgrade/test_upgrade: from_kafka_version=0.9.0.1, 
> to_message_format_version=0.9.0.1, compression_types=.lz4
> {code:java}
> 3 acked message did not make it to the Consumer. They are: [33906, 33900, 
> 33903]. The first 3 missing messages were validated to ensure they are in 
> Kafka's data files. 3 were missing. This suggests data loss. Here are some of 
> the messages not found in the data files: [33906, 33900, 33903]
> Traceback (most recent call last):
>   File 
> "/home/jenkins/workspace/system-test-kafka_5.3.x/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/tests/runner_client.py",
>  line 132, in run
>     data = self.run_test()
>   File 
> "/home/jenkins/workspace/system-test-kafka_5.3.x/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/tests/runner_client.py",
>  line 189, in run_test
>     return self.test_context.function(self.test)
>   File 
> "/home/jenkins/workspace/system-test-kafka_5.3.x/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/mark/_mark.py",
>  line 428, in wrapper
>     return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/home/jenkins/workspace/system-test-kafka_5.3.x/kafka/tests/kafkatest/tests/core/upgrade_test.py",
>  line 136, in test_upgrade
>     self.run_produce_consume_validate(core_test_action=lambda: 
> self.perform_upgrade(from_kafka_version,
>   File 
> "/home/jenkins/workspace/system-test-kafka_5.3.x/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 112, in run_produce_consume_validate
>     self.validate()
>   File 
> "/home/jenkins/workspace/system-test-kafka_5.3.x/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 135, in validate
>     assert succeeded, error_msg
> AssertionError: 3 acked message did not make it to the Consumer. They are: 
> [33906, 33900, 33903]. The first 3 missing messages were validated to ensure 
> they are in Kafka's data files. 3 were missing. This suggests data loss. Here 
> are some of the messages not found in the data files: [33906, 33900, 33903]
> {code}
> Logs show:
>  # Broker 1 is leader of partition
>  # Broker 2 successfully fetches from offset 10947 and processes request
>  # Broker 2 sends fetch request to broker 1 for offset 10950
>  # Broker 1 sets is HW to 10950, acknowledges produce requests up to HW
>  # Broker 2 is elected leader
>  # Broker 2 truncates to its local HW of 10947 - 3 messages are lost
> This data loss is a known issue that was fixed under KIP-101. But since this 
> can still happen with older messages formats, we should update upgrade tests 
> to cope with some data loss.
>   



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Build failed in Jenkins: kafka-0.10.2-jdk7 #243

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8722; Ensure record CRC is always verified when appending to log

--
[...truncated 1.12 MB...]
org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCantAllocateMoreMemoryThanWeHave STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCantAllocateMoreMemoryThanWeHave PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnBlockTimeout STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnBlockTimeout PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testSimple STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testSimple PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testDelayedAllocation STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testDelayedAllocation PASSED

org.apache.kafka.clients.producer.ProducerRecordTest > testInvalidRecords 
STARTED

org.apache.kafka.clients.producer.ProducerRecordTest > testInvalidRecords PASSED

org.apache.kafka.clients.producer.ProducerRecordTest > testEqualsAndHashCode 
STARTED

org.apache.kafka.clients.producer.ProducerRecordTest > testEqualsAndHashCode 
PASSED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse STARTED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse PASSED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest PASSED

org.apache.kafka.clients.NetworkClientTest > testClose STARTED

org.apache.kafka.clients.NetworkClientTest > testClose PASSED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode STARTED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected 
STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected PASSED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout STARTED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithNoBrokerDiscovery STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithNoBrokerDiscovery PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay PASSED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode STARTED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayDisconnected 
STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayDisconnected 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > 
testUsableVersionCalculationNoKnownVersions STARTED

org.apache.kafka.clients.NodeApiVersionsTest > 
testUsableVersionCalculationNoKnownVersions PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testVersionsToString STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testVersionsToString PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUnsupportedVersionsToString 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUnsupportedVersionsToString 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUnknownApiVersionsToString 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUnknownApiVersionsToString 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionCalculation 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionCalculation 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionLatestVersions 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionLatestVersions 
PASSED

> Task :clients:determineCommitId UP-TO-DATE
> Task :clients:createVersionFile
> Task :clients:jar UP-TO-DATE
> Task :core:compileJava NO-SOURCE

> Task :core:compileScala
Pruning sources from previous analysis, due to incompatible CompileSetup.
:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DE

Re: [DISCUSS] KIP-499 - Unify connection name flag for command line tool

[Mitchell]

Hi Jason,
Thanks for looking at this!

I wasn't exactly sure what to put in the compatibility section.  I wrote
the KIP thinking that we should probably mark the old arguments for
deprecation for a release or two before actually removing them.  I'm happy
to change this either way if it's preferred.
-mitch

On Tue, Jul 30, 2019 at 11:55 PM Jason Gustafson  wrote:

> Hey Mitch, thanks for the KIP! This command line inconsistency frustrates
> me almost every day. I'm definitely +1 on this.
>
> One minor nitpick. The compatibility section mentions there will be no
> deprecations, but it sounds like we are planning on deprecating the old
> arguments?
>
> Thanks,
> Jason
>
> On Tue, Jul 30, 2019 at 5:25 PM Mitchell  wrote:
>
> > Hello,
> > I have written a proposal to add the command line argument
> > `--bootstrap-server` to 5 of the existing command line tools that do not
> > currently use `--broker-list` for passing cluster connection information.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool
> >
> > Please take a look and let me know what you think.
> > Thanks,
> > -Mitch
> >
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Mickael Maison]

Hi Justine,

Thanks for the response!
In my opinion, it would be better if the producer did not rely at all
on the broker auto create feature as this is what we're aiming to
deprecate. When requesting metadata we can set the
"allow_auto_topic_creation" field to false to avoid the broker auto
creation. Then if the topic is not existing, send a
CreateTopicRequest.

What do you think?

On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan  wrote:
>
> Currently the way it is implemented, the broker auto-creation configuration
> takes precedence. The producer will not use the CreateTopics request.
> (Technically it can--but the topic will already be created through the
> broker, so it will never try to create the topic.)
> It is possible to change this however, and I'd be happy to discuss the
> benefits of this alternative.
>
> Thank you,
> Justine
>
> On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison 
> wrote:
>
> > Hi Justine,
> >
> > Thanks for the KIP!
> >
> > In case auto creation is enabled on both the client and server, will
> > the producer still use the AdminClient (CreateTopics request) to
> > create topics? and not rely on the broker auto create.
> > I'm guessing the answer is yes but can you make it explicit.
> >
> > Thank you
> >
> > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan 
> > wrote:
> > >
> > > Hi,
> > > Just a friendly reminder to take a look at this KIP if you have the time.
> > >
> > > I was thinking about broker vs. client default precedence, and I think it
> > > makes sense to keep the broker as the default used when both client-side
> > > and broker-side defaults are configured. The idea is that there would be
> > > pretty clear documentation, and that many systems with configurations
> > that
> > > the client could not change would likely have the auto-create default
> > off.
> > > (In cloud for example).
> > >
> > > It also seems like in most cases, the consumer config
> > > 'allow.auto.create.topics' was created to actually prevent the creation
> > of
> > > topics, so the loss of creation functionality will not be a big problem.
> > >
> > >  I'm happy to discuss any other compatibility problems or components of
> > > this KIP.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan 
> > wrote:
> > >
> > > > Hello all,
> > > >
> > > > I was looking at this KIP again, and there is a decision I made that I
> > > > think is worth discussing.
> > > >
> > > > In the case where both the broker and producer's
> > > > 'auto.create.topics.enable' are set to true, we have to choose either
> > the
> > > > broker configs or the producer configs for the replication
> > > > factor/partitions.
> > > >
> > > > Currently, the decision is to have the broker defaults take precedence.
> > > > (It is easier to do this in the implementation.) It also makes some
> > sense
> > > > for this behavior to take precedence since this behavior already
> > occurs as
> > > > the default.
> > > >
> > > > However, I was wondering if it would be odd for those who can only see
> > the
> > > > producer side to set configs for replication factor/partitions and see
> > > > different results. Currently the documentation for the config states
> > that
> > > > the config values are only used when the broker config is not enabled,
> > but
> > > > this might not always be clear to the user.  Changing the code to have
> > the
> > > > producer's configurations take precedence is possible, but I was
> > wondering
> > > > what everyone thought.
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan 
> > > > wrote:
> > > >
> > > >> Just a quick update--
> > > >>
> > > >> It seems that enabling both the broker and producer configs works
> > fine,
> > > >> except that the broker configurations for partitions, replication
> > factor
> > > >> take precedence.
> > > >> I don't know if that is something we would want to change, but I'll be
> > > >> updating the KIP for now to reflect this. Perhaps we would want to
> > add more
> > > >> to the documentation of the the producer configs to clarify.
> > > >>
> > > >> Thank you,
> > > >> Justine
> > > >>
> > > >> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan 
> > > >> wrote:
> > > >>
> > > >>> Hi Colin,
> > > >>>
> > > >>> Thanks for looking at the KIP. I can definitely add to the title to
> > make
> > > >>> it more clear.
> > > >>>
> > > >>> It makes sense that both configurations could be turned on since
> > there
> > > >>> are many cases where the user can not control the server-side
> > > >>> configurations. I was a little concerned about how both interacting
> > would
> > > >>> work out -- if there would be to many requests for new topics, for
> > example.
> > > >>> But it since it does make sense to allow both configurations
> > enabled, I
> > > >>> will test out some scenarios and I'll change the KIP to support this.
> > > >>>
> > > >>> I also agree with documentation about distinguishing the
> > differ

Build failed in Jenkins: kafka-trunk-jdk8 #3820

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Refactor abstractConfig#configuredInstance (#7129)

--
[...truncated 2.57 MB...]
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTaskPut STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testHandleExceptionInTaskPut PASSED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInTaskPut STARTED

org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperatorTest > 
testThrowExceptionInTaskPut PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogOnDisabledLogReporter STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogOnDisabledLogReporter PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogOnEnabledLogReporter STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogOnEnabledLogReporter PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogMessageWithNoRecords STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogMessageWithNoRecords PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogMessageWithSinkRecords STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testLogMessageWithSinkRecords PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > testSetDLQConfigs 
STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > testSetDLQConfigs 
PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDlqHeaderConsumerRecord STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDlqHeaderConsumerRecord PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDlqHeaderOnNullExceptionMessage STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDlqHeaderOnNullExceptionMessage PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDlqHeaderIsAppended STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDlqHeaderIsAppended PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
initializeDLQWithNullMetrics STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
initializeDLQWithNullMetrics PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDLQConfigWithEmptyTopicName STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDLQConfigWithEmptyTopicName PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDLQConfigWithValidTopicName STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > 
testDLQConfigWithValidTopicName PASSED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > testReportDLQTwice 
STARTED

org.apache.kafka.connect.runtime.errors.ErrorReporterTest > testReportDLQTwice 
PASSED

org.apache.kafka.connect.runtime.WorkerTaskTest > 
updateMetricsOnListenerEventsForStartupPauseResumeAndShutdown STARTED

org.apache.kafka.connect.runtime.WorkerTaskTest > 
updateMetricsOnListenerEventsForStartupPauseResumeAndShutdown PASSED

org.apache.kafka.connect.runtime.WorkerTaskTest > standardStartup STARTED

org.apache.kafka.connect.runtime.WorkerTaskTest > standardStartup PASSED

org.apache.kafka.connect.runtime.WorkerTaskTest > stopBeforeStarting STARTED

org.apache.kafka.connect.runtime.WorkerTaskTest > stopBeforeStarting PASSED

org.apache.kafka.connect.runtime.WorkerTaskTest > cancelBeforeStopping STARTED

org.apache.kafka.connect.runtime.WorkerTaskTest > cancelBeforeStopping PASSED

org.apache.kafka.connect.runtime.WorkerTaskTest > 
updateMetricsOnListenerEventsForStartupPauseResumeAndFailure STARTED

org.apache.kafka.connect.runtime.WorkerTaskTest > 
updateMetricsOnListenerEventsForStartupPauseResumeAndFailure PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariable STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariable PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTL STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTL PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLAndScheduleRestart STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLAndScheduleRestart PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLFirstCancelThenScheduleRestart STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testReplaceVariableWithTTLFirstCancelThenScheduleRestart PASSED

org.apache.kafka.connect.runtime.WorkerConfigTransformerTest > 
testTransformNullConfiguration STARTED

org.apache.kafka.connect.runtime.WorkerConfigTransforme

Re: [ANNOUNCE] Apache Kafka 2.3.0

[Mickael Maison]

Hi,

It looks like the protocol page was not updated. It still only lists 2.2 APIs.
http://kafka.apache.org/protocol

Thanks

On Tue, Jul 2, 2019 at 2:05 AM Colin McCabe  wrote:
>
> Hi Mickael,
>
> Thanks for pointing this out.  It should be fixed now.
>
> best,
> Colin
>
> On Mon, Jul 1, 2019, at 09:14, Mickael Maison wrote:
> > Colin,
> >
> > The javadocs links are broken:
> > The requested URL /23/javadoc/index.html was not found on this server.
> >
> > It's the 3rd time in a row this happens (2.1 and 2.2 had the same
> > issue at release). Last time, Guozhang confirmed this step is in the
> > release process but maybe this needs to be highlighted
> >
> > On Tue, Jun 25, 2019 at 8:22 PM Colin McCabe  wrote:
> > >
> > > Thanks to everyone who reviewed the Apache blog post about 2.3.  It's 
> > > live now at https://blogs.apache.org/kafka/date/20190624
> > >
> > > Plus, Tim Berglund made a video about what's new in this release.  
> > > https://www.youtube.com/watch?v=sNqwJT2WguQ
> > >
> > > Finally, check out Stéphane Maarek's video about 2.3 here: 
> > > https://www.youtube.com/watch?v=YutjYKSGd64
> > >
> > > cheers,
> > > Colin
> > >
> > >
> > > On Tue, Jun 25, 2019, at 09:40, Colin McCabe wrote:
> > > > The Apache Kafka community is pleased to announce the release for
> > > > Apache Kafka 2.3.0.
> > > > This release includes several new features, including:
> > > >
> > > > - There have been several improvements to the Kafka Connect REST API.
> > > > - Kafka Connect now supports incremental cooperative rebalancing.
> > > > - Kafka Streams now supports an in-memory session store and window
> > > > store.
> > > > - The AdminClient now allows users to determine what operations they
> > > > are authorized to perform on topics.
> > > > - There is a new broker start time metric.
> > > > - JMXTool can now connect to secured RMI ports.
> > > > - An incremental AlterConfigs API has been added.  The old AlterConfigs
> > > > API has been deprecated.
> > > > - We now track partitions which are under their min ISR count.
> > > > - Consumers can now opt-out of automatic topic creation, even when it
> > > > is enabled on the broker.
> > > > - Kafka components can now use external configuration stores (KIP-421)
> > > > - We have implemented improved replica fetcher behavior when errors are
> > > > encountered
> > > >
> > > > All of the changes in this release can be found in the release notes:
> > > > https://www.apache.org/dist/kafka/2.3.0/RELEASE_NOTES.html
> > > >
> > > >
> > > > You can download the source and binary release (Scala 2.11 and 2.12) 
> > > > from:
> > > > https://kafka.apache.org/downloads#2.3.0
> > > >
> > > > All of the changes in this release can be found in the release notes:
> > > > https://www.apache.org/dist/kafka/2.3.0/RELEASE_NOTES.html
> > > >
> > > > You can download the source and binary release (Scala 2.11 and 2.12) 
> > > > from:
> > > > https://kafka.apache.org/downloads#2.3.0
> > > >
> > > > ---
> > > >
> > > > Apache Kafka is a distributed streaming platform with four core APIs:
> > > >
> > > > ** The Producer API allows an application to publish a stream records
> > > > to one or more Kafka topics.
> > > >
> > > > ** The Consumer API allows an application to subscribe to one or more
> > > > topics and process the stream of records produced to them.
> > > >
> > > > ** The Streams API allows an application to act as a stream processor,
> > > > consuming an input stream from one or more topics and producing an
> > > > output stream to one or more output topics, effectively transforming
> > > > the input streams to output streams.
> > > >
> > > > ** The Connector API allows building and running reusable producers or
> > > > consumers that connect Kafka topics to existing applications or data
> > > > systems. For example, a connector to a relational database might
> > > > capture every change to a table.
> > > >
> > > >
> > > > With these APIs, Kafka can be used for two broad classes of application:
> > > > ** Building real-time streaming data pipelines that reliably get data
> > > > between systems or applications.
> > > > ** Building real-time streaming applications that transform or react to
> > > > the streams of data.
> > > >
> > > > Apache Kafka is in use at large and small companies worldwide,
> > > > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > > > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > > > Zalando, among others.
> > > >
> > > > A big thank you for the following 101 contributors to this release!
> > > >
> > > > Aishwarya Gune, Alex Diachenko, Alex Dunayevsky, Anna Povzner, Arabelle
> > > > Hou, Arjun Satish, A. Sophie Blee-Goldman, asutosh936, Bill Bejeck, Bob
> > > > Barrett, Boyang Chen, Brian Bushree, cadonna, Casey Green, Chase
> > > > Walden, Chia-Ping Tsai, Chris Egerton, Chris Steingen, Colin Hicks,
> > > > Colin Patrick McCabe,

Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM

[Gokul Ramanan Subramanian]

Hi Alex.

A rejected alternative is to try to do SSL handshake from the plaintext 
transport layer. This couples various transport layers and is inflexible to 
adding new transport layers in the future. Further, if the plaintext transport 
layer does SSL handshake, it will always get an error, irrespective of whether 
or not the peer supports SSL. Therefore, the plaintext transport layer would 
have to determine if the peer uses SSL or not based on the type of error it 
gets back from the SSL handshake. This fits right into the general anti-pattern 
of using exceptions as control flow mechanisms.

Another rejected alternative would be to have a single byte at the transport 
layer represent the security protocol in use. This byte would have to be 
present in every single message between clients and brokers and between brokers 
and brokers. This change is backwards-incompatible and adds overhead to Kafka. 
It is likely never going to get accepted by the community.

In the absence of a fool-proof way to do a handhskake across all security 
protocols (plaintext, SSL, other future ones), I think that the proposed KIP 
provides a good solution. Therefore, you have my +1. (Not sure if my +1 counts, 
since I am a Kafka noob).

Thanks.
Gokul Ramanan Subramanian
Senior SDE, Amazon AWS

On 28/Jul/19 05:43:19PM +0100, Alexandre Dupriez wrote:
 Hello,
 
 I have created the KIP-498
 

 to
 propose a minor addition to the producer configuration properties, in order
 to protect against OOM when the response message decoded by a client
 indicates an "abnormally high" size to be allocated.
 
 This follows this discussion thread
  and is
 tracked by KAFKA-4090 .
 
 Please let me know what you think. I created this KIP even though the
 change seems minimal because it affects client configuration, which is
 mentioned as a type of change eligible for a KIP on this main wiki page
 
 .
 
 Thanks,
 Alexandre

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

[Stanislav Kozlovski]

Hey Jun,

I think I can answer some of your questions on behalf of Colin -- he can
confirm if I'm correct

> 10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to
LeaderAndIsr request. Could you explain how these 2 fields will be used?
Sorry for not explaining this in the KIP - those fields won't be used by
the non-controller brokers yet. Our plans for them are outlined in the
Future Work section of the KIP - namely "Reassignment Quotas that only
throttle reassignment traffic" and "Add reassignment metrics".

> Should we include those two fields in UpdateMetadata and potentially
Metadata requests too?
I recall this was discussed in the beginning by Colin and Jason, so I'll
let Colin answer that question.

11 & 12. Correct, we need to send StopReplica requests. The implementation
does this (
https://github.com/apache/kafka/pull/7128/files#diff-ed90e8ecc5439a5ede5e362255d11be1R651)
-- I'll update the KIP to mention it as well.
I tried to document the algorithm here
https://github.com/apache/kafka/pull/7128/files#diff-ed90e8ecc5439a5ede5e362255d11be1R521
.

13. I think so. (
https://github.com/apache/kafka/pull/7128#discussion_r308866206). I'll
reflect this in the KIP

Here is the updated KIP diff -
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=112820260&selectedPageVersions=36&selectedPageVersions=35

Thanks,
Stanislav

On Tue, Jul 30, 2019 at 10:18 PM Jun Rao  wrote:

> Hi, Colin,
>
> Thanks for the KIP. Sorry for the late reply. LGTM overall. A few detailed
> comments below.
>
> 10. The KIP adds two new fields (AddingReplicas and RemovingReplicas) to
> LeaderAndIsr request. Could you explain how these 2 fields will be used?
> Should we include those two fields in UpdateMetadata and potentially
> Metadata requests too?
>
> 11. "If a new reassignment is issued during an on-going one, we cancel the
> current one by emptying out both AR and RR, constructing them from (the
> updated from the last-reassignment) R and TR, and starting anew." In this
> case, it seems that the controller needs to issue a StopReplica request to
> remove those unneeded replicas.
>
> 12. "Essentially, once a cancellation is called we subtract AR from R,
> empty out both AR and RR, and send LeaderAndIsr requests to cancel the
> replica movements that have not yet completed." Similar to the above, it
> seems the controller needs to issue a StopReplica request to remove those
> unneeded replicas.
>
> 13. Since we changed the format of the topics/[topic] zNode, should we bump
> up the version number in the json value?
>
> Jun
>
> On Mon, Jul 22, 2019 at 8:38 AM Colin McCabe  wrote:
>
> > Hi all,
> >
> > With three non-binding +1 votes from Viktor Somogyi-Vass, Robert Barrett,
> > and George Li, and 3 binding +1 votes from Gwen Shapira, Jason Gustafson,
> > and myself, the vote passes.  Thanks, everyone!
> >
> > best,
> > Colin
> >
> > On Fri, Jul 19, 2019, at 08:55, Robert Barrett wrote:
> > > +1 (non-binding). Thanks for the KIP!
> > >
> > > On Thu, Jul 18, 2019 at 5:59 PM George Li  > .invalid>
> > > wrote:
> > >
> > > >  +1 (non-binding)
> > > >
> > > >
> > > >
> > > > Thanks for addressing the comments.
> > > > George
> > > >
> > > > On Thursday, July 18, 2019, 05:03:58 PM PDT, Gwen Shapira <
> > > > g...@confluent.io> wrote:
> > > >
> > > >  Renewing my +1, thank you Colin and Stan for working through all the
> > > > questions, edge cases, requests and alternatives. We ended up with a
> > > > great protocol.
> > > >
> > > > On Thu, Jul 18, 2019 at 4:54 PM Jason Gustafson 
> > > > wrote:
> > > > >
> > > > > +1 Thanks for the KIP. Really looking forward to this!
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Jul 17, 2019 at 1:41 PM Colin McCabe 
> > wrote:
> > > > >
> > > > > > Thanks, Stanislav.  Let's restart the vote to reflect the fact
> that
> > > > we've
> > > > > > made significant changes.  The new vote will go for 3 days as
> > usual.
> > > > > >
> > > > > > I'll start with my +1 (binding).
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 17, 2019, at 08:56, Stanislav Kozlovski wrote:
> > > > > > > Hey everybody,
> > > > > > >
> > > > > > > We have further iterated on the KIP in the accompanying
> > discussion
> > > > thread
> > > > > > > and I'd like to propose we resume the vote.
> > > > > > >
> > > > > > > Some notable changes:
> > > > > > > - we will store reassignment information in the
> > > > `/brokers/topics/[topic]`
> > > > > > > - we will internally use two collections to represent a
> > reassignment
> > > > -
> > > > > > > "addingReplicas" and "removingReplicas". LeaderAndIsr has been
> > > > updated
> > > > > > > accordingly
> > > > > > > - the Alter API will still use the "targetReplicas" collection,
> > but
> > > > the
> > > > > > > List API will now return three separate collections - the full
> > > > replica
> > > > > > set,
> > > > > > > the replicas we are adding as part of this reassignment
> > > > > > 

[jira] [Resolved] (KAFKA-8722) Data crc check repair

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8722.

   Resolution: Fixed
Fix Version/s: (was: 0.10.2.2)
   0.10.2.3

> Data crc check repair
> -
>
> Key: KAFKA-8722
> URL: https://issues.apache.org/jira/browse/KAFKA-8722
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.10.2.2
>Reporter: ChenLin
>Priority: Major
> Fix For: 0.10.2.3
>
> Attachments: image-2019-07-27-14-50-08-128.png, 
> image-2019-07-27-14-50-58-300.png, image-2019-07-27-14-56-25-610.png, 
> image-2019-07-27-14-57-06-687.png, image-2019-07-27-15-05-12-565.png, 
> image-2019-07-27-15-06-07-123.png, image-2019-07-27-15-10-21-709.png, 
> image-2019-07-27-15-18-22-716.png, image-2019-07-30-11-39-01-605.png
>
>
> In our production environment, when we consume kafka's topic data in an 
> operating program, we found an error:
> org.apache.kafka.common.KafkaException: Record for partition 
> rl_dqn_debug_example-49 at offset 2911287689 is invalid, cause: Record is 
> corrupt (stored crc = 3580880396, computed crc = 1701403171)
>  at 
> org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:869)
>  at 
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:788)
>  at 
> org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:480)
>  at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1188)
>  at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1046)
>  at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:88)
>  at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
>  at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
>  at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
>  at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> At this point we used the kafka.tools.DumpLogSegments tool to parse the disk 
> log file and found that there was indeed dirty data:
> !image-2019-07-27-14-57-06-687.png!
> By looking at the code, I found that in some cases kafka would not verify the 
> data and write it to disk, so we fixed it.
>  We found that when record.offset is not equal to the offset we are 
> expecting, kafka will set the variable inPlaceAssignment to false. When 
> inPlaceAssignment is false, data will not be verified:
> !image-2019-07-27-14-50-58-300.png!
> !image-2019-07-27-14-50-08-128.png!
> Our repairs are as follows:
> !image-2019-07-30-11-39-01-605.png!
> We did a comparative test for this. By modifying the client-side producer 
> code, we made some dirty data. For the original kafka version, it was able to 
> write to the disk normally, but when it was consumed, it was reported, but 
> our repaired version was written. At the time, it can be verified, so this 
> producer write failed:
> !image-2019-07-27-15-05-12-565.png!
> At this time, when the client consumes, an error will be reported:
> !image-2019-07-27-15-06-07-123.png!
> When the kafka server is replaced with the repaired version, the producer 
> will verify that the dirty data is written. The producer failed to write the 
> data this time
> !image-2019-07-27-15-10-21-709.png!



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[VOTE] KIP-373: Allow users to create delegation tokens for other users

[Viktor Somogyi-Vass]

Hi All,

I'd like to start a vote on this KIP.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users

To summarize it: the proposed feature would allow users (usually
superusers) to create delegation tokens for other users. This is especially
helpful in Spark where the delegation token created this way can be
distributed to workers.

I'd be happy to receive any votes or additional feedback.

Viktor

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

[Viktor Somogyi-Vass]

Hi Folks,

I'm starting a vote on this.

Viktor

On Thu, Jun 27, 2019 at 12:02 PM Viktor Somogyi-Vass <
viktorsomo...@gmail.com> wrote:

> Hi Folks,
>
> I took over this issue from Manikumar. Recently another motivation have
> been raised in Spark for this (SPARK-28173) and I think it'd be great to
> continue this task.
> I updated the KIP and will wait for a few days to get some feedback then
> proceed for the vote.
>
> Thanks,
> Viktor
>
> On Tue, Dec 11, 2018 at 8:29 AM Manikumar 
> wrote:
>
>> Hi Harsha,
>>
>> Thanks for the review.
>>
>> With this KIP a designated superuser can create tokens without requiring
>> individual user credentials.
>> Any client can authenticate brokers using the created tokens. We may not
>> call this as impersonation,
>> since the clients API calls are executing on their own authenticated
>> connections.
>>
>> Thanks,
>> Manikumar
>>
>> On Fri, Dec 7, 2018 at 11:56 PM Harsha  wrote:
>>
>> > Hi Mani,
>> >  Overall KIP looks good to me. Can we call this
>> Impersonation
>> > support, which is what the KIP is doing?
>> > Also instead of using super.uses as the config which essentially giving
>> > cluster-wide support to the users, we can introduce impersonation.users
>> as
>> > a config and users listed in the config are allowed to impersonate other
>> > users.
>> >
>> > Thanks,
>> > Harsha
>> >
>> >
>> > On Fri, Dec 7, 2018, at 3:58 AM, Manikumar wrote:
>> > > Bump up! to get some attention.
>> > >
>> > > BTW, recently Apache Spark added  support for Kafka delegation token.
>> > > https://issues.apache.org/jira/browse/SPARK-25501
>> > >
>> > > On Fri, Dec 7, 2018 at 5:27 PM Manikumar 
>> > wrote:
>> > >
>> > > > Bump up! to get some attention.
>> > > >
>> > > > BTW, recently Apache Spark added for Kafka delegation token support.
>> > > > https://issues.apache.org/jira/browse/SPARK-25501
>> > > >
>> > > > On Tue, Sep 25, 2018 at 9:56 PM Manikumar <
>> manikumar.re...@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> I have created a KIP that proposes to allow users to create
>> delegation
>> > > >> tokens for other users.
>> > > >>
>> > > >>
>> > > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
>> > > >>
>> > > >> Please take a look when you get a chance.
>> > > >>
>> > > >> Thanks,
>> > > >> Manikumar
>> > > >>
>> > > >
>> >
>>
>

[jira] [Created] (KAFKA-8739) rejoining broker fails to sanity check existing log segments

[sanjiv marathe (JIRA)]

sanjiv marathe created KAFKA-8739:
-

 Summary: rejoining broker fails to sanity check existing log 
segments
 Key: KAFKA-8739
 URL: https://issues.apache.org/jira/browse/KAFKA-8739
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 2.3.0
Reporter: sanjiv marathe


kafka claims it can be used as a storage. But following scenario proves other 
wise.
 # Consider a topic with single partition, repl-factor 2, with two brokers, say 
A and B with A is the leader.
 # Broker B fails due to sector errors. Sysadmin fixes the issues and brings it 
up again after a few minutes. A few log segments are lost/corrupted.
 # Broker B catches up with missed out msgs by fetching them from the leader A, 
but does not realize the issue with earlier log segments.
 # Broker A fails, B becomes the leader.
 # A new consumer requests msgs from the beginning. Broker B fails to deliver 
msgs belonging to corrupted log segments.

Suggested solution

A broker, immediately after coming up, should go through a sanity check, e.g. 
CRC check of its log segments. Any corrupted/lost, should be refetched from the 
leader.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)