Re: [DISCUSS] : KIP-562: Allow fetching a key from a single partition rather than iterating over all the stores on an instance

[Matthias J. Sax]

Thanks for the KIP. Overall it makes sense.

Couple of minor comments/questions:

10) To me, it was initially quite unclear why we need this KIP. The
motivation section does only talk about some performance issues (that
are motivated by single key look-ups) -- however, all issues mentioned
in the KIP could be fixed without any public API change. The important
cases, why the public API changes (and thus this KIP) is useful are
actually missing in the motivation section. I would be helpful to add
more details.

20) `StoreQueryParams` has a lot of getter methods that we usually don't
have for config objects (compare `Consumed`, `Produced`, `Materialized`,
etc). Is there any reason why we need to add those getters to the public
API?

30) The change to remove `KafkaStreams#store(...)` as introduced in
KIP-535 should be listed in sections Public API changes. Also, existing
methods should not be listed -- only changes. Hence, in
`KafkaStreams.java` only one new method and the `store()` method as
added via KIP-535 should be listed.

40) `QueryableStoreProvider` and `StreamThreadStateStoreProvider` are
internal classes and thus we can remove all changes to it from the KIP.


Thanks!


-Matthias



On 1/21/20 11:46 AM, Vinoth Chandar wrote:
> Chiming in a bit late here..
> 
> +1 This is a very valid improvement. Avoiding doing gets on irrelevant
> partitions will improve performance and efficiency for IQs.
> 
> As an incremental improvement to the current APIs,  adding an option to
> filter out based on partitions makes sense
> 
> 
> 
> 
> 
> 
> 
> On Mon, Jan 20, 2020 at 3:13 AM Navinder Brar
>  wrote:
> 
>> Thanks John. If there are no other comments to be addressed, I will start
>> a vote today so that we are on track for this release.~Navinder
>>
>>
>> On Monday, January 20, 2020, 8:32 AM, John Roesler 
>> wrote:
>>
>> Thanks, Navinder,
>>
>> The Param object looks a bit different than I would have done, but it
>> certainly is explicit. We might have to deprecate those particular factory
>> methods and move to a builder pattern if we need to add any more options in
>> the future, but I’m fine with that possibility.
>>
>> The KIP also discusses some implementation details that aren’t necessary
>> here. We really only need to see the public interfaces. We can discuss the
>> implementation in the PR.
>>
>> That said, the public API part of the current proposal looks good to me! I
>> would be a +1 if you called for a vote.
>>
>> Thanks,
>> John
>>
>> On Sun, Jan 19, 2020, at 20:50, Navinder Brar wrote:
>>> I have made some edits in the KIP, please take another look. It would
>>> be great if we can push it in 2.5.0.
>>> ~Navinder
>>>
>>>
>>> On Sunday, January 19, 2020, 12:59 AM, Navinder Brar
>>>  wrote:
>>>
>>> Sure John, I will update the StoreQueryParams with static factory
>>> methods.
>>> @Ted, we would need to create taskId only in case a user provides one
>>> single partition. In case user wants to query all partitions of an
>>> instance the current code is good enough where we iterate over all
>>> stream threads and go over all taskIds to match the store. But in case
>>> a user requests for a single partition-based store, we need to create a
>>> taskId out of that partition and store name(using
>>> internalTopologyBuilder class) and match with the taskIds belonging to
>>> that instance. I will add the code in the KIP.
>>>
>>> On Sunday, 19 January, 2020, 12:47:08 am IST, Ted Yu
>>>  wrote:
>>>
>>>  Looking at the current KIP-562:
>>>
>>> bq. Create a taskId from the combination of store name and partition
>>> provided by the user
>>>
>>> I wonder if a single taskId would be used for the “all partitions” case.
>>> If so, we need to choose a numerical value for the partition portion of
>> the
>>> taskId.
>>>
>>> On Sat, Jan 18, 2020 at 10:27 AM John Roesler 
>> wrote:
>>>
 Thanks, Ted!

 This makes sense, but it seems like we should lean towards explicit
 semantics in the public API. ‘-1’ meaning “all partitions” is
>> reasonable,
 but not explicit. That’s why I suggested the Boolean for “all
>> partitions”.
 I guess this also means getPartition() should either throw an
>> exception or
 return null if the partition is unspecified.

 Thanks,
 John

 On Sat, Jan 18, 2020, at 08:43, Ted Yu wrote:
> I wonder if the following two methods can be combined:
>
> Integer getPartition() // would be null if unset or if "all
>> partitions"
> boolean getAllLocalPartitions() // true/false if "all partitions"
 requested
>
> into:
>
> Integer getPartition() // would be null if unset or -1 if "all
 partitions"
>
> Cheers
>
> On Fri, Jan 17, 2020 at 9:56 PM John Roesler 
 wrote:
>
>> Thanks, Navinder!
>>
>> I took a look at the KIP.
>>
>> We tend to use static factory methods instead of public
>> constructors,
 and
>> also builders for optional parameters.
>>
>> Given that, I 

Enable both SASL & SSL authentication...

[Senthilnathan Muthusamy]

Hi,

We want both SASL (oAuthBearer) & SSL authentication to be enabled. However 
based on the below doc, the SSL auth will be disabled if SASL is enabled.

https://docs.confluent.io/current/kafka/authentication_ssl.html#brokers


If any SASL authentication mechanisms are enabled for a given listener, then 
SSL client authentication is disabled-even if you have specified 
ssl.client.auth=required and the broker authenticates clients only using SASL 
on that listener.

How can we have both SASL & SSL authentication enabled as some of our clients 
uses oAuth and some uses cert based auth?

Appreciate any pointers.

Thanks,
Senthil

Build failed in Jenkins: kafka-trunk-jdk11 #1101

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Only update a request's local complete time in API handler if


--
[...truncated 5.75 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

Build failed in Jenkins: kafka-trunk-jdk8 #4177

[Apache Jenkins Server]

See 


Changes:

[jason] MINOR: Only update a request's local complete time in API handler if


--
[...truncated 5.73 MB...]

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task :streams:upgrade-system-tests-0102:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0102:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0102:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0102:compileTestJava
> Task :streams:upgrade-system-tests-0102:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0102:testClasses
> Task 

Build failed in Jenkins: kafka-2.2-jdk8 #21

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-9143: Log task reconfiguration error only when it happened 
(#7648)


--
[...truncated 2.04 MB...]
kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters STARTED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsDefaultValueIfNotExist 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsDefaultValueIfNotExist 
PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter PASSED

kafka.utils.CommandLineUtilsTest > 
testMaybeMergeOptionsDefaultOverwriteExisting STARTED

kafka.utils.CommandLineUtilsTest > 
testMaybeMergeOptionsDefaultOverwriteExisting PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsNotOverwriteExisting 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsNotOverwriteExisting 
PASSED

kafka.utils.JsonTest > testParseToWithInvalidJson STARTED

kafka.utils.JsonTest > testParseToWithInvalidJson PASSED

kafka.utils.JsonTest > testParseTo STARTED

kafka.utils.JsonTest > testParseTo PASSED

kafka.utils.JsonTest > testJsonParse STARTED

kafka.utils.JsonTest > testJsonParse PASSED

kafka.utils.JsonTest > testLegacyEncodeAsString STARTED

kafka.utils.JsonTest > testLegacyEncodeAsString PASSED

kafka.utils.JsonTest > testEncodeAsBytes STARTED

kafka.utils.JsonTest > testEncodeAsBytes PASSED

kafka.utils.JsonTest > testEncodeAsString STARTED

kafka.utils.JsonTest > testEncodeAsString PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod STARTED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions STARTED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.PasswordEncoderTest > testEncoderConfigChange STARTED

kafka.utils.PasswordEncoderTest > testEncoderConfigChange PASSED

kafka.utils.PasswordEncoderTest > testEncodeDecodeAlgorithms STARTED

kafka.utils.PasswordEncoderTest > testEncodeDecodeAlgorithms PASSED

kafka.utils.PasswordEncoderTest > testEncodeDecode STARTED

kafka.utils.PasswordEncoderTest > testEncodeDecode PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

Re: [DISCUSS] KIP-501 Avoid out-of-sync or offline partitions when follower fetch requests not processed in time

[Satish Duggana]

Hi Jun,
Can you please review the KIP and let us know your comments?

If there are no comments/questions, we can start a vote thread.

It looks like Yelp folks also encountered the same issue as mentioned
in JIRA comment[1].

>> Flavien Raynaud added a comment - Yesterday
We've seen offline partitions happening for the same reason in one of
our clusters too, where only the broker leader for the offline
partitions was having disk issues. It looks like there has not been
much progress/look on the PR submitted since December 9th. Is there
anything blocking this change from moving forward?

1. 
https://issues.apache.org/jira/browse/KAFKA-8733?focusedCommentId=17020083=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17020083

Thanks,
Satish.


On Thu, Dec 5, 2019 at 10:38 AM Harsha Chintalapani  wrote:
>
> Hi Jason,
>  As Satish said just increase replica max lag will not work in this
> case. Just before a disk dies the reads becomes really slow and its hard to
> estimate how much this is, as we noticed range is pretty wide. Overall it
> doesn't make sense to knock good replicas out of just because a leader is
> slower in processing reads or serving the fetch requests which may be due
> to disk issues in this case but could be other issues as well. I think this
> kip addresses in general all of these issues.
>  Do you still have questions on the current approach if not we can
> take it vote.
> Thanks,
> Harsha
>
>
> On Mon, Nov 18, 2019 at 7:05 PM, Satish Duggana 
> wrote:
>
> > Hi Jason,
> > Thanks for looking into the KIP. Apologies for my late reply. Increasing
> > replica max lag to 30-45 secs did not help as we observed that a few fetch
> > requests took more than 1-2 minutes. We do not want to increase further as
> > it increases upper bound on commit latency. We have strict SLAs on some of
> > the clusters on end to end(producer to consumer) latency. This proposal
> > improves the availability of partitions when followers are trying their
> > best to be insync even when leaders are slow in processing those requests.
> > I have updated the KIP to have a single config for giving backward
> > compatibility and I guess this config is more comprehensible than earlier.
> > But I believe there is no need to have config because the suggested
> > proposal in the KIP is an enhancement to the existing behavior. Please let
> > me know your comments.
> >
> > Thanks,
> > Satish.
> >
> > On Thu, Nov 14, 2019 at 10:57 AM Jason Gustafson 
> > wrote:
> >
> > Hi Satish,
> >
> > Thanks for the KIP. I'm wondering how much of this problem can be
> > addressed just by increasing the replication max lag? That was one of the
> > purposes of KIP-537 (the default increased from 10s to 30s). Also, the new
> > configurations seem quite low level. I think they will be hard for users to
> > understand (even reading through a couple times I'm not sure I understand
> > them fully). I think if there's a way to improve this behavior without
> > requiring any new configurations, it would be much more attractive.
> >
> > Best,
> > Jason
> >
> > On Wed, Nov 6, 2019 at 8:14 AM Satish Duggana 
> > wrote:
> >
> > Hi Dhruvil,
> > Thanks for looking into the KIP.
> >
> > 10. I have an initial sketch of the KIP-500 in commit[a] which discusses
> > tracking the pending fetch requests. Tracking is not done in
> > Partition#readRecords because if it takes longer in reading any of the
> > partitions then we do not want any of the replicas of this fetch request to
> > go out of sync.
> >
> > 11. I think `Replica` class should be thread-safe to handle the remote
> > scenario of concurrent requests running for a follower replica. Or I may be
> > missing something here. This is a separate issue from KIP-500. I will file
> > a separate JIRA to discuss that issue.
> >
> > a -
> > https://github.com/satishd/kafka/commit/
> > c69b525abe8f6aad5059236076a003cdec4c4eb7
> >
> > Thanks,
> > Satish.
> >
> > On Tue, Oct 29, 2019 at 10:57 AM Dhruvil Shah 
> > wrote:
> >
> > Hi Satish,
> >
> > Thanks for the KIP, those seems very useful. Could you elaborate on how
> > pending fetch requests are tracked?
> >
> > Thanks,
> > Dhruvil
> >
> > On Mon, Oct 28, 2019 at 9:43 PM Satish Duggana  >
> > wrote:
> >
> > Hi All,
> > I wrote a short KIP about avoiding out-of-sync or offline partitions when
> > follower fetch requests are not processed in time by the leader replica.
> > KIP-501 is located at https://s.apache.org/jhbpn
> >
> > Please take a look, I would like to hear your feedback and suggestions.
> >
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-8733
> >
> > Thanks,
> > Satish.
> >
> >

Build failed in Jenkins: kafka-2.4-jdk8 #130

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-9083: Various fixes/improvements for Connect's Values class

[rhauch] KAFKA-9024: Better error message when field specified does not exist


--
[...truncated 5.49 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

[jira] [Created] (KAFKA-9463) Transient failure in KafkaAdminClientTest.testListOffsets

[Ted Yu (Jira)]

Ted Yu created KAFKA-9463:
-

 Summary: Transient failure in KafkaAdminClientTest.testListOffsets
 Key: KAFKA-9463
 URL: https://issues.apache.org/jira/browse/KAFKA-9463
 Project: Kafka
  Issue Type: Test
Reporter: Ted Yu


When running tests with Java 11, I got the following test failure:
{code}
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node 
assignment.
at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at 
org.apache.kafka.clients.admin.KafkaAdminClientTest.testListOffsets(KafkaAdminClientTest.java:2336)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting 
for a node assignment.
{code}
KafkaAdminClientTest.testListOffsets passes when it is run alone.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-trunk-jdk11 #1100

[Apache Jenkins Server]

See 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Thanks Matthias.

The KIP is about InvalidStateStoreException.
I pick `StateStoreNotAvailableException` because it may be more intuitive
than `StreamsNotRunningException`.

No matter which one picked, it's good to me.

---
Vito


On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax 
wrote:

> Thanks for updating the KIP!
>
> One last comment/question: you kept `StateStoreNotAvailableException` in
> favor of `StreamsNotRunningException` (to merge both as suggested).
>
> I am wondering, if it might be better to keep
> `StreamsNotRunningException` instead of
> `StateStoreNotAvailableException`, because this exception is thrown if
> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
>
>
>
> -Matthias
>
> On 1/17/20 9:56 PM, John Roesler wrote:
> > Thanks, Vito. I've just cast my vote.
> > -John
> >
> > On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >> Hi, folks,
> >>
> >> Just update the KIP, please take a look.
> >>
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng  wrote:
> >>
> >>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> >>> I got a lot out of the discussion.
> >>>
> >>> I would like to update KIP-216 base on John's suggestion to remove the
> >>> category.
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax  >
> >>> wrote:
> >>>
> > Nevertheless, if we omit the categorization, it’s moot.
> 
>  Ack.
> 
>  I am fine to remove the middle tier. As John pointed out, it might be
>  weird to have only one concrete exception type per category. We can
> also
>  explain in detail how to handle each exception in their JavaDocs.
> 
> 
>  -Matthias
> 
>  On 1/16/20 6:38 AM, Bill Bejeck wrote:
> > Vito,
> >
> > Thanks for the updates, the KIP LGTM.
> >
> > -Bill
> >
> > On Wed, Jan 15, 2020 at 11:31 PM John Roesler 
>  wrote:
> >
> >> Hi Vito,
> >>
> >> Haha, your archive game is on point!
> >>
> >> What Matthias said in that email is essentially what I figured was
> the
> >> rationale. It makes sense, but the point I was making is that this
>  really
> >> doesn’t seem like a good way to structure a production app. On the
>  other
> >> hand, considering the exception fatal has a good chance of avoiding
> a
> >> frustrating debug session if you just forgot to call start.
> >>
> >> Nevertheless, if we omit the categorization, it’s moot.
> >>
> >> It would be easy to add a categorization layer later if we want it,
> but
> >> not very easy to change it if we get it wrong.
> >>
> >> Thanks for your consideration!
> >> -John
> >>
> >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >>> Hi John,
> >>>
> >>> About `StreamsNotStartedException is strange` --
> >>> The original idea came from Matthias, two years ago. :)
> >>> You can reference here:
> >>>
> >>
> 
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e
> >>>
> >>> About omitting the categorization --
> >>> It looks reasonable. I'm fine with omitting the categorization but
> not
> >> very
> >>> sure it is a good choice.
> >>> Does any other folks provide opinion?
> >>>
> >>>
> >>> Hi, folks,
> >>>
> >>> Just update the KIP-216, please take a look.
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng 
>  wrote:
> >>>
> 
>  Hi, folks,
> 
>  Thank you suggestion, really appreciate it. :)
>  I understand your concern. I'll merge StreamsNotRunningException
> and
>  StateStoreNotAvailableException.
> 
> 
>  ---
>  Vito
> 
> 
>  On Thu, Jan 16, 2020 at 6:22 AM John Roesler  >
> >> wrote:
> 
> > Hey Vito,
> >
> > Yes, thanks for the KIP. Sorry the discussion has been so long.
> > Hopefully, we can close it out soon.
> >
> > I agree we can drop StreamsNotRunningException in favor of
> > just StateStoreNotAvailableException.
> >
> > Unfortunately, I have some higher-level concerns. The value
> > of these exceptions is that they tell you how to handle the
> > various situations that can arise while querying a distributed
> > data store.
> >
> > Ideally, as a caller, I should be able to just catch "retriable"
> or
> > "fatal" and handle them appropriately. Otherwise, there's no
> > point in having categories, and we should just have all the
> > exceptions extend InvalidStateStoreException.
> >
> > Presently, it's not possible to tell from just the
> > "retriable"/"fatal" distinction what to do. You  can 

Build failed in Jenkins: kafka-trunk-jdk8 #4176

[Apache Jenkins Server]

See 


Changes:

[rhauch] MINOR: log error message from Connect sink exception (#7555)

[rhauch] KAFKA-9083: Various fixes/improvements for Connect's Values class

[wangguoz] KAFKA-7204: Avoid clearing records for paused partitions on poll of

[rhauch] KAFKA-9024: Better error message when field specified does not exist

[jason] MINOR: Change the log level from ERROR to DEBUG when failing to get

[rhauch] KAFKA-9143: Log task reconfiguration error only when it happened 
(#7648)


--
[...truncated 5.72 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task 

Build failed in Jenkins: kafka-2.2-jdk8 #20

[Apache Jenkins Server]

See 


Changes:

[rhauch] MINOR: Embedded connect cluster should mask exit procedures by default

[rhauch] KAFKA-9083: Various fixes/improvements for Connect's Values class

[rhauch] KAFKA-9024: Better error message when field specified does not exist


--
[...truncated 2.06 MB...]
kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters STARTED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsDefaultValueIfNotExist 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsDefaultValueIfNotExist 
PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter PASSED

kafka.utils.CommandLineUtilsTest > 
testMaybeMergeOptionsDefaultOverwriteExisting STARTED

kafka.utils.CommandLineUtilsTest > 
testMaybeMergeOptionsDefaultOverwriteExisting PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsNotOverwriteExisting 
STARTED

kafka.utils.CommandLineUtilsTest > testMaybeMergeOptionsNotOverwriteExisting 
PASSED

kafka.utils.JsonTest > testParseToWithInvalidJson STARTED

kafka.utils.JsonTest > testParseToWithInvalidJson PASSED

kafka.utils.JsonTest > testParseTo STARTED

kafka.utils.JsonTest > testParseTo PASSED

kafka.utils.JsonTest > testJsonParse STARTED

kafka.utils.JsonTest > testJsonParse PASSED

kafka.utils.JsonTest > testLegacyEncodeAsString STARTED

kafka.utils.JsonTest > testLegacyEncodeAsString PASSED

kafka.utils.JsonTest > testEncodeAsBytes STARTED

kafka.utils.JsonTest > testEncodeAsBytes PASSED

kafka.utils.JsonTest > testEncodeAsString STARTED

kafka.utils.JsonTest > testEncodeAsString PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod STARTED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions STARTED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.PasswordEncoderTest > testEncoderConfigChange STARTED

kafka.utils.PasswordEncoderTest > testEncoderConfigChange PASSED

kafka.utils.PasswordEncoderTest > testEncodeDecodeAlgorithms STARTED

kafka.utils.PasswordEncoderTest > testEncodeDecodeAlgorithms PASSED

kafka.utils.PasswordEncoderTest > testEncodeDecode STARTED

kafka.utils.PasswordEncoderTest > testEncodeDecode PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > 

[jira] [Created] (KAFKA-9462) Correct exception message in DistributedHerder

[Ted Yu (Jira)]

Ted Yu created KAFKA-9462:
-

 Summary: Correct exception message in DistributedHerder
 Key: KAFKA-9462
 URL: https://issues.apache.org/jira/browse/KAFKA-9462
 Project: Kafka
  Issue Type: Bug
Reporter: Ted Yu


There are a few exception messages in DistributedHerder which were copied from 
other exception message.

This task corrects the messages to reflect actual condition



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-2.2-jdk8-old #201

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-9143: Log task reconfiguration error only when it happened 
(#7648)


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H35 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.2^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.2^{commit} # timeout=10
Checking out Revision f480bf67783d672d21251904bcd98087735b24fa 
(refs/remotes/origin/2.2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f480bf67783d672d21251904bcd98087735b24fa
Commit message: "KAFKA-9143: Log task reconfiguration error only when it 
happened (#7648)"
 > git rev-list --no-walk b4b35c6d0ac4ee714137235e7adcbca9d89be2d0 # timeout=10
ERROR: No tool found matching GRADLE_4_8_1_HOME
[kafka-2.2-jdk8-old] $ /bin/bash -xe /tmp/jenkins1288317200919645671.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins1288317200919645671.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
ERROR: No tool found matching GRADLE_4_8_1_HOME
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
ERROR: No tool found matching GRADLE_4_8_1_HOME
No credentials specified
ERROR: No tool found matching GRADLE_4_8_1_HOME
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=f480bf67783d672d21251904bcd98087735b24fa, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #175
Recording test results
ERROR: No tool found matching GRADLE_4_8_1_HOME
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
ERROR: No tool found matching GRADLE_4_8_1_HOME
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user ism...@juma.me.uk
Not sending mail to unregistered user j...@confluent.io
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user b...@confluent.io

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Matthias J. Sax]

Thanks for updating the KIP!

One last comment/question: you kept `StateStoreNotAvailableException` in
favor of `StreamsNotRunningException` (to merge both as suggested).

I am wondering, if it might be better to keep
`StreamsNotRunningException` instead of
`StateStoreNotAvailableException`, because this exception is thrown if
Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?



-Matthias

On 1/17/20 9:56 PM, John Roesler wrote:
> Thanks, Vito. I've just cast my vote.
> -John
> 
> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>> Hi, folks,
>>
>> Just update the KIP, please take a look.
>>
>> Thanks!
>>
>> ---
>> Vito
>>
>>
>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng  wrote:
>>
>>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
>>> I got a lot out of the discussion.
>>>
>>> I would like to update KIP-216 base on John's suggestion to remove the
>>> category.
>>>
>>>
>>> ---
>>> Vito
>>>
>>>
>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax 
>>> wrote:
>>>
> Nevertheless, if we omit the categorization, it’s moot.

 Ack.

 I am fine to remove the middle tier. As John pointed out, it might be
 weird to have only one concrete exception type per category. We can also
 explain in detail how to handle each exception in their JavaDocs.


 -Matthias

 On 1/16/20 6:38 AM, Bill Bejeck wrote:
> Vito,
>
> Thanks for the updates, the KIP LGTM.
>
> -Bill
>
> On Wed, Jan 15, 2020 at 11:31 PM John Roesler 
 wrote:
>
>> Hi Vito,
>>
>> Haha, your archive game is on point!
>>
>> What Matthias said in that email is essentially what I figured was the
>> rationale. It makes sense, but the point I was making is that this
 really
>> doesn’t seem like a good way to structure a production app. On the
 other
>> hand, considering the exception fatal has a good chance of avoiding a
>> frustrating debug session if you just forgot to call start.
>>
>> Nevertheless, if we omit the categorization, it’s moot.
>>
>> It would be easy to add a categorization layer later if we want it, but
>> not very easy to change it if we get it wrong.
>>
>> Thanks for your consideration!
>> -John
>>
>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>>> Hi John,
>>>
>>> About `StreamsNotStartedException is strange` --
>>> The original idea came from Matthias, two years ago. :)
>>> You can reference here:
>>>
>>
 https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e
>>>
>>> About omitting the categorization --
>>> It looks reasonable. I'm fine with omitting the categorization but not
>> very
>>> sure it is a good choice.
>>> Does any other folks provide opinion?
>>>
>>>
>>> Hi, folks,
>>>
>>> Just update the KIP-216, please take a look.
>>>
>>> ---
>>> Vito
>>>
>>>
>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng 
 wrote:
>>>

 Hi, folks,

 Thank you suggestion, really appreciate it. :)
 I understand your concern. I'll merge StreamsNotRunningException and
 StateStoreNotAvailableException.


 ---
 Vito


 On Thu, Jan 16, 2020 at 6:22 AM John Roesler 
>> wrote:

> Hey Vito,
>
> Yes, thanks for the KIP. Sorry the discussion has been so long.
> Hopefully, we can close it out soon.
>
> I agree we can drop StreamsNotRunningException in favor of
> just StateStoreNotAvailableException.
>
> Unfortunately, I have some higher-level concerns. The value
> of these exceptions is that they tell you how to handle the
> various situations that can arise while querying a distributed
> data store.
>
> Ideally, as a caller, I should be able to just catch "retriable" or
> "fatal" and handle them appropriately. Otherwise, there's no
> point in having categories, and we should just have all the
> exceptions extend InvalidStateStoreException.
>
> Presently, it's not possible to tell from just the
> "retriable"/"fatal" distinction what to do. You  can tell
> from the descriptions of the various exceptions. E.g.:
>
> Retriable:
>  * StreamsRebalancingException: the exact same call
> should just be retried until the rebalance is complete
>  * StateStoreMigratedException: the store handle is
> now invalid, so you need to re-discover the instance
> and get a new handle on that instance. In other words,
> the query itself may be valid, but the particular method
> invocation on this particular instance has encountered
> a fatal 

Build failed in Jenkins: kafka-trunk-jdk11 #1099

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9457; Fix flaky test


--
[...truncated 5.75 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 

Build failed in Jenkins: kafka-2.3-jdk8 #165

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-9143: Log task reconfiguration error only when it happened 
(#7648)


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H34 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.3^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.3^{commit} # timeout=10
Checking out Revision d92192192d4e18ff32325e12c02b7b0226eb2a98 
(refs/remotes/origin/2.3)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d92192192d4e18ff32325e12c02b7b0226eb2a98
Commit message: "KAFKA-9143: Log task reconfiguration error only when it 
happened (#7648)"
 > git rev-list --no-walk 781f53496f9a20fde46e19671a69eae5ac45e8fe # timeout=10
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
[kafka-2.3-jdk8] $ /bin/bash -xe /tmp/jenkins4753128026073637686.sh
+ rm -rf 
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
[kafka-2.3-jdk8] $ /bin/bash -xe /tmp/jenkins389918031343184562.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew --no-daemon -PmaxParallelForks=1 
-PtestLoggingEvents=started,passed,skipped,failed -PxmlFindBugsReport=true 
clean testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/5.4.1/userguide/gradle_daemon.html.

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the User Manual chapter on the daemon at 
https://docs.gradle.org/5.4.1/userguide/gradle_daemon.html
Process command line: /usr/local/asfpackages/java/jdk1.8.0_191/bin/java -Xss2m 
-Xmx1024m -Dfile.encoding=ISO-8859-1 -Duser.country=US -Duser.language=en 
-Duser.variant -cp 
/home/jenkins/.gradle/wrapper/dists/gradle-5.4.1-all/3221gyojl5jsh0helicew7rwx/gradle-5.4.1/lib/gradle-launcher-5.4.1.jar
 org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.4.1
Please read the following process output to find out more:
---
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /home/jenkins/.gradle/daemon/5.4.1/hs_err_pid11695.log


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
No credentials specified
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=d92192192d4e18ff32325e12c02b7b0226eb2a98, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #151
Recording test results
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user j...@confluent.io
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user b...@confluent.io

Build failed in Jenkins: kafka-trunk-jdk8 #4175

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9457; Fix flaky test


--
[...truncated 5.71 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task 

Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[Gwen Shapira]

Thank you for the KIP. Awesomely cloud-native improvement :)

+1 (binding)


On Tue, Jan 21, 2020, 9:35 AM David Jacot  wrote:

> Hi all,
>
> I would like to start a vote on KIP-559: Make the Kafka Protocol Friendlier
> with L7 Proxies.
>
> The KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies
>
> Thanks,
> David
>

Build failed in Jenkins: kafka-2.3-jdk8 #164

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-9083: Various fixes/improvements for Connect's Values class

[rhauch] KAFKA-9024: Better error message when field specified does not exist


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H34 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.3^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.3^{commit} # timeout=10
Checking out Revision 781f53496f9a20fde46e19671a69eae5ac45e8fe 
(refs/remotes/origin/2.3)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 781f53496f9a20fde46e19671a69eae5ac45e8fe
Commit message: "KAFKA-9024: Better error message when field specified does not 
exist (#7819)"
 > git rev-list --no-walk 9d17521e0e177bcdd6aa8843dc3c8af38d58330f # timeout=10
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
[kafka-2.3-jdk8] $ /bin/bash -xe /tmp/jenkins8862954192381455386.sh
+ rm -rf 
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
[kafka-2.3-jdk8] $ /bin/bash -xe /tmp/jenkins1928506141352492051.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew --no-daemon -PmaxParallelForks=1 
-PtestLoggingEvents=started,passed,skipped,failed -PxmlFindBugsReport=true 
clean testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/5.4.1/userguide/gradle_daemon.html.

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the User Manual chapter on the daemon at 
https://docs.gradle.org/5.4.1/userguide/gradle_daemon.html
Process command line: /usr/local/asfpackages/java/jdk1.8.0_191/bin/java -Xss2m 
-Xmx1024m -Dfile.encoding=ISO-8859-1 -Duser.country=US -Duser.language=en 
-Duser.variant -cp 
/home/jenkins/.gradle/wrapper/dists/gradle-5.4.1-all/3221gyojl5jsh0helicew7rwx/gradle-5.4.1/lib/gradle-launcher-5.4.1.jar
 org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.4.1
Please read the following process output to find out more:
---
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /home/jenkins/.gradle/daemon/5.4.1/hs_err_pid4618.log


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
No credentials specified
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=781f53496f9a20fde46e19671a69eae5ac45e8fe, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #151
Recording test results
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_10_3_HOME=/home/jenkins/tools/gradle/4.10.3
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user j...@confluent.io
Not sending mail to unregistered user 

Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

[KUN DU]

Rebased the PR based on latest trunk, can we merge the PR?

Kun

radai  于2019年12月13日周五 上午9:26写道：

> i also have a PR :-)
> https://github.com/apache/kafka/pull/7569
>
> On Thu, Dec 12, 2019 at 9:50 PM Gwen Shapira  wrote:
> >
> > You got 3 binding votes (Joel, Harsha, Ismael) - the vote passed on Nov
> 7.
> >
> > Happy hacking!
> >
> > On Thu, Dec 12, 2019 at 11:35 AM radai 
> wrote:
> > >
> > > so can we call this passed ?
> > >
> > > On Thu, Nov 7, 2019 at 7:43 AM Satish Duggana <
> satish.dugg...@gmail.com> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > On Thu, Nov 7, 2019 at 8:58 PM Ismael Juma 
> wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Thu, Oct 24, 2019 at 9:33 PM radai 
> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'd like to initiate a vote on KIP-514.
> > > > > >
> > > > > > links:
> > > > > > the kip -
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer
> > > > > > the PR - https://github.com/apache/kafka/pull/7569
> > > > > >
> > > > > > Thank you
> > > > > >
>

Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[David Arthur]

David, thanks for the KIP. This looks like a nice improvement. Explicit
over implicit :)

+1, binding

-David

On Tue, Jan 21, 2020 at 4:17 PM Jason Gustafson  wrote:

> Thanks, +1
>
> On Tue, Jan 21, 2020 at 1:06 PM David Jacot  wrote:
>
> > Hi Jason,
> >
> > Thanks for your feedback. You have brought a good point. I did not
> consider
> > this but your are right. It makes sense to add it as well in order to
> able
> > to fully
> > parse the SyncGroup request/response. Let me update the KIP.
> >
> > Thanks,
> > David
> >
> > On Tue, Jan 21, 2020 at 12:52 PM Jason Gustafson 
> > wrote:
> >
> > > Hi David,
> > >
> > > Just one question. In addition to including the protocol type, I'm
> > > wondering if there is value in adding the protocol name to SyncGroup?
> > This
> > > would potentially give you the ability to parse the "user data" field
> in
> > > the consumer group schema.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Tue, Jan 21, 2020 at 9:35 AM David Jacot 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to start a vote on KIP-559: Make the Kafka Protocol
> > > Friendlier
> > > > with L7 Proxies.
> > > >
> > > > The KIP is here:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies
> > > >
> > > > Thanks,
> > > > David
> > > >
> > >
> >
>


-- 
David Arthur

Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[Jason Gustafson]

Thanks, +1

On Tue, Jan 21, 2020 at 1:06 PM David Jacot  wrote:

> Hi Jason,
>
> Thanks for your feedback. You have brought a good point. I did not consider
> this but your are right. It makes sense to add it as well in order to able
> to fully
> parse the SyncGroup request/response. Let me update the KIP.
>
> Thanks,
> David
>
> On Tue, Jan 21, 2020 at 12:52 PM Jason Gustafson 
> wrote:
>
> > Hi David,
> >
> > Just one question. In addition to including the protocol type, I'm
> > wondering if there is value in adding the protocol name to SyncGroup?
> This
> > would potentially give you the ability to parse the "user data" field in
> > the consumer group schema.
> >
> > Thanks,
> > Jason
> >
> > On Tue, Jan 21, 2020 at 9:35 AM David Jacot  wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a vote on KIP-559: Make the Kafka Protocol
> > Friendlier
> > > with L7 Proxies.
> > >
> > > The KIP is here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies
> > >
> > > Thanks,
> > > David
> > >
> >
>

Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[David Jacot]

Hi Jason,

Thanks for your feedback. You have brought a good point. I did not consider
this but your are right. It makes sense to add it as well in order to able
to fully
parse the SyncGroup request/response. Let me update the KIP.

Thanks,
David

On Tue, Jan 21, 2020 at 12:52 PM Jason Gustafson  wrote:

> Hi David,
>
> Just one question. In addition to including the protocol type, I'm
> wondering if there is value in adding the protocol name to SyncGroup? This
> would potentially give you the ability to parse the "user data" field in
> the consumer group schema.
>
> Thanks,
> Jason
>
> On Tue, Jan 21, 2020 at 9:35 AM David Jacot  wrote:
>
> > Hi all,
> >
> > I would like to start a vote on KIP-559: Make the Kafka Protocol
> Friendlier
> > with L7 Proxies.
> >
> > The KIP is here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies
> >
> > Thanks,
> > David
> >
>

Build failed in Jenkins: kafka-2.3-jdk8 #163

[Apache Jenkins Server]

See 

Changes:


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H34 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress -- 
https://github.com/apache/kafka.git +refs/heads/*:refs/remotes/origin/*" 
returned status code 128:
stdout: 
stderr: error: cannot create thread: Resource temporarily unavailable
fatal: fetch-pack: unable to fork off sideband demultiplexer

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:758)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to 
H34
at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at 
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor1081.invoke(Unknown 
Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy152.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1152)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at 
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at 
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at 
hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
ERROR: Error cloning remote repo 'origin'
Retrying after 10 seconds
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/kafka.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at 

Re: [VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[Jason Gustafson]

Hi David,

Just one question. In addition to including the protocol type, I'm
wondering if there is value in adding the protocol name to SyncGroup? This
would potentially give you the ability to parse the "user data" field in
the consumer group schema.

Thanks,
Jason

On Tue, Jan 21, 2020 at 9:35 AM David Jacot  wrote:

> Hi all,
>
> I would like to start a vote on KIP-559: Make the Kafka Protocol Friendlier
> with L7 Proxies.
>
> The KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies
>
> Thanks,
> David
>

Build failed in Jenkins: kafka-2.2-jdk8-old #200

[Apache Jenkins Server]

See 


Changes:

[rhauch] MINOR: Embedded connect cluster should mask exit procedures by default

[rhauch] KAFKA-9083: Various fixes/improvements for Connect's Values class

[rhauch] KAFKA-9024: Better error message when field specified does not exist


--
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on H35 (ubuntu) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress -- https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.2^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.2^{commit} # timeout=10
Checking out Revision b4b35c6d0ac4ee714137235e7adcbca9d89be2d0 
(refs/remotes/origin/2.2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b4b35c6d0ac4ee714137235e7adcbca9d89be2d0
Commit message: "KAFKA-9024: Better error message when field specified does not 
exist (#7819)"
 > git rev-list --no-walk c2b98560a20e8fdc17d7586ccaf64e1fd60f334d # timeout=10
ERROR: No tool found matching GRADLE_4_8_1_HOME
[kafka-2.2-jdk8-old] $ /bin/bash -xe /tmp/jenkins6736948535966574886.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins6736948535966574886.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
ERROR: No tool found matching GRADLE_4_8_1_HOME
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
ERROR: No tool found matching GRADLE_4_8_1_HOME
No credentials specified
ERROR: No tool found matching GRADLE_4_8_1_HOME
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=b4b35c6d0ac4ee714137235e7adcbca9d89be2d0, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #175
Recording test results
ERROR: No tool found matching GRADLE_4_8_1_HOME
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
ERROR: No tool found matching GRADLE_4_8_1_HOME
Not sending mail to unregistered user wangg...@gmail.com
Not sending mail to unregistered user ism...@juma.me.uk
Not sending mail to unregistered user j...@confluent.io
Not sending mail to unregistered user nore...@github.com
Not sending mail to unregistered user b...@confluent.io

[jira] [Resolved] (KAFKA-9024) org.apache.kafka.connect.transforms.ValueToKey throws NPE

[Randall Hauch (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-9024.
--
  Reviewer: Randall Hauch
Resolution: Fixed

Merged to `trunk` and backported to the `2.4`, `2.3`, and `2.2` branches, since 
we typically backport only to the last 2-3 branches.

> org.apache.kafka.connect.transforms.ValueToKey throws NPE
> -
>
> Key: KAFKA-9024
> URL: https://issues.apache.org/jira/browse/KAFKA-9024
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Robin Moffatt
>Assignee: Nigel Liang
>Priority: Minor
> Fix For: 2.2.3, 2.5.0, 2.3.2, 2.4.1
>
>
> If a field named in the SMT does not exist a NPE is thrown. This is not 
> helpful to users and should be caught correctly and reported back in a more 
> friendly way.
> For example, importing data from a database with this transform: 
>  
> {code:java}
> transforms = [ksqlCreateKey, ksqlExtractString]
> transforms.ksqlCreateKey.fields = [ID]
> transforms.ksqlCreateKey.type = class 
> org.apache.kafka.connect.transforms.ValueToKey
> transforms.ksqlExtractString.field = ID
> transforms.ksqlExtractString.type = class 
> org.apache.kafka.connect.transforms.ExtractField$Key
> {code}
> If the field name is {{id}} not {{ID}} then the task fails : 
> {code:java}
> org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error 
> handler
>at 
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
>at 
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
>at 
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:50)
>at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:293)
>at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:229)
>at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
>at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
>at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
>at 
> org.apache.kafka.connect.transforms.ValueToKey.applyWithSchema(ValueToKey.java:85)
>at org.apache.kafka.connect.transforms.ValueToKey.apply(ValueToKey.java:65)
>at 
> org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50)
>at 
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
>at 
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
>... 11 more
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Mickael Maison]

+1 (binding)
Thanks

On Tue, Jan 21, 2020 at 7:58 PM Ron Dagostino  wrote:
>
> +1 (non-binding)
>
> Ron
>
> On Tue, Jan 21, 2020 at 11:29 AM Manikumar  wrote:
> >
> > +1 (binding).
> >
> > Thanks for the KIP.
> >
> >
> > On Tue, Jan 21, 2020 at 9:56 PM Ted Yu  wrote:
> >
> > > +1
> > >
> > > On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram 
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks for the KIP!
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков 
> > > > wrote:
> > > >
> > > > > Hello.
> > > > >
> > > > > I would like to start vote for KIP-553: Disable all SSL protocols
> > > except
> > > > > TLSV1.2 by default.
> > > > >
> > > > > KIP -
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> > > > > Discussion thread -
> > > > >
> > > >
> > > https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
> > > >
> > >

Re: [VOTE] KIP-558: Add Connect REST API endpoints to view the topics used by connectors in Kafka Connect

[Bill Bejeck]

Thanks for the KIP Konstantine.  This will be very useful for Connect.

+1(binding)

-Bill

On Tue, Jan 21, 2020 at 2:12 PM Almog Gavra  wrote:

> Another thanks from me! +1 (non-binding)
>
> On Tue, Jan 21, 2020 at 11:04 AM Randall Hauch  wrote:
>
> > Thanks again for the KIP and this improvement for Connect.
> >
> > +1 (binding)
> >
> > Randall
> >
> > On Tue, Jan 21, 2020 at 10:45 AM Tom Bentley 
> wrote:
> >
> > > +1 (non-binding). Thanks for the KIP Konstantine.
> > >
> > > On Sat, Jan 18, 2020 at 2:18 AM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to open the vote on KIP-558 that had a constructive flurry
> of
> > > > discussions in the past few days, in order to give this KIP the
> > > opportunity
> > > > to be voted on by the current KIP deadline (Wed, Jan 22, 2020), if -
> of
> > > > course - there's agreement upon its final form.
> > > >
> > > > KIP link here:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
> > > >
> > > > Best regards,
> > > > Konstantine
> > > >
> > >
> >
>

[jira] [Resolved] (KAFKA-9083) Various parsing issues in Values class

[Randall Hauch (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-9083.
--
  Reviewer: Randall Hauch
Resolution: Fixed

Merged to the `trunk`, `2.4`, `2.3`, and `2.2` branches; we typically don't 
push bugfixes back further than 2-3 branches.

> Various parsing issues in Values class
> --
>
> Key: KAFKA-9083
> URL: https://issues.apache.org/jira/browse/KAFKA-9083
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 2.2.3, 2.5.0, 2.3.2, 2.4.1
>
>
> There are a number of small issues with the Connect framework's {{Values}} 
> class that lead to either unexpected exceptions, unintuitive (and arguably 
> incorrect) behavior, or confusing log messages. These include:
>  * A {{NullPointerException}} is thrown when parsing the string {{[null]}} 
> (which should be parsed as an array containing a single null element)
>  * A {{NullPointerException}} is thrown when parsing the string {{[]}} (which 
> should be parsed as an empty array)
>  * The returned schema is null when parsing the string {{{}}} (instead, it 
> should be a map schema, possibly with null key and value schemas)
>  * Strings that begin with what appear to be booleans (i.e., the literals 
> {{true}} or {{false}}) and which are followed by token delimiters (e.g., {{}, 
> {{]}}, {{:}}, etc.) are parsed as booleans when they should arguably be 
> parsed as strings; for example, the string {{true}}} is parsed as the boolean 
> {{true}} but should probably just be parsed as the string {{true}}}
>  * Arrays not containing commas are still parsed as arrays in some cases; for 
> example, the string {{[0 1 2]}} is parsed as the array {{[0, 1, 2]}} when it 
> should arguably be parsed as the string literal {{[0 1 2]}}
>  * An exception is logged when attempting to parse input as a map when it 
> doesn't contain the a final closing brace that states "Map is missing 
> terminating ']'" even though the expected terminating character is actually 
> {{}} and not {{]}}
>  * Finally, and possibly most contentious, escape sequences are not stripped 
> from string literals. Thus, the input string 
> {{foobar]}} is parsed as the literal string 
> {{foobar]}}, which is somewhat unexpected, since 
> that means it is impossible to pass in a string that is parsed as the string 
> literal {{foobar]}}, and it is the job of the caller to handle 
> stripping of such escape sequences. Given that the escape sequence can itself 
> be escaped, it seems better to automatically strip it from parsed string 
> literals before returning them to the user.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-546: Add quota-specific APIs to the Admin Client, redux

[Jason Gustafson]

Hi Brian,

Thanks for the proposal! I have a couple comments/questions:

1. I'm having a hard time understanding the point of `QuotaEntity.Type`. It
sounds like this might be just for convenience since the APIs are using
string types. If so, I think it's a bit misleading to represent it as an
enum. In particular, I cannot see how the UNKNOWN type would be used. The
`PrincipalBuilder` plugin allows users to provide their own principal type,
so I think the API should be usable even for unknown entity types. Note
also that we appear to be relying on this enum in `QuotaFilter` class. I
think that should be changed to just a string?

2. It's a little annoying that we have two separate APIs to describe client
quotas. The names do not really make it clear which API someone should use.
It might just be a naming problem. In the command utility, it looks like
you are using --list and --describe to distinguish the two. Perhaps the
APIs can be named similarly: e.g. ListClientQuotas and
DescribeClientQuotas. However, looking at the examples, it's still not very
clear to me why we need both options. Basically I'm finding the
"config-centric" mode not very intuitive.

Thanks,
Jason


On Fri, Jan 17, 2020 at 2:14 PM Brian Byrne  wrote:

> Thanks Colin, I've updated the KIP with the relevant changes.
>
> On Fri, Jan 17, 2020 at 10:17 AM Colin McCabe  wrote:
>
> > I thought about this a little bit more, and maybe we can leave in the
> > enums rather than going with strings.  But we need to have an "UNKNOWN"
> > value for all the enums, so that if a value that the client doesn't
> > understand is returned, it can get translated to that.  This is what we
> did
> > with the ACLs API, and it worked out well.
> >
>
> Done. One thing I omitted here was that the API still accepts/returns
> Strings, since there may be plugins that specify their own types/units. If
> we'd like to keep it this way, then the UNKNOWN may be unnecessary. Let me
> know how you'd feel this is best resolved.
>
>
> > On balance, I think we should leave in "units."  It could be useful for
> > future-proofing.
> >
>
> Done. Also added a comment in the ClientQuotaCommand to default to RATE_BPS
> if no unit is supplied to ease adoption.
>
>
> > Also, since there are other kinds of quotas not covered by this API, we
> > should rename DescribeQuotas -> DescribeClientQuotas, AlterQuotas ->
> > AlterClientQuotas, etc. etc.
> >
>
> Done. Updated command and script name, too.
>
>
> > Maybe QuotaFilter doesn't need a "rule" argument to its constructor right
> > now.  We can just do literal matching for everything.  Like I said
> earlier,
> > I don't think people do a lot of prefixing of principal names.  When we
> > added the "prefix matching" stuff for ACLs, it was mostly to let people
> do
> > it for topics.  Then we made it more generic because it was easy to do
> so.
> > In this case, the API is probably easier to understand if we just do a
> > literal match.  We can always have a follow-on KIP to add fancier
> filtering
> > if needed.
> >
>
> Done.
>
>
> > For DescribeEffectiveQuotasResult, if you request all relevant quotas, it
> > would be nice to see which ones apply and which ones don't.  Right now,
> you
> > just get a map, but you don't know which quotas are actually in force,
> and
> > which are not relevant but might be in the future if a different quota
> gets
> > deleted.  One way to do this would be to have two maps-- one for
> applicable
> > quotas and one for shadowed quotas.
> >
>
> So the way it's specified is that it maps QuotaKey -> Value, however Value
> is actually defined to have two parts: the entry, and a list of overridden
> entries (where an entry is the value, along with the source). Perhaps the
> Value is poorly named, or maybe there's a simpler structure to be had?
>
> Thanks,
> Brian
>
>
>
> > On Tue, Jan 14, 2020, at 13:32, Brian Byrne wrote:
> > > Hi Colin,
> > >
> > > Your feedback is appreciated, thank you.
> > >
> > > On Tue, Jan 14, 2020 at 11:34 AM Colin McCabe 
> > wrote:
> > >
> > > > This is probably a nitpick, but it would be nice to specify that this
> > list
> > > > is in order of highest priority to lowest.
> > > >
> > >
> > > Done.
> > >
> > >
> > > > Hmm.  Maybe --show-overridden or --include-overridden is a better
> flag
> > > > name?
> > > >
> > >
> > > Done (--show-overridden).
> > >
> > >
> > > > I think it would be nice to avoid using enums for QuotaEntity#Type,
> > > > QuotaKey#Type, and QuotaFilter#Rule.  With enums, we have to worry
> > about
> > > > forwards and backwards compatibility problems.  For example, what do
> > you do
> > > > when you're querying a broker that has a new value for one of these,
> > that
> > > > is not in your enum?  In the  past, we've created an UNKNOWN value
> for
> > enum
> > > > types to solve this conundrum, but I'm not sure the extra complexity
> is
> > > > worth it here.  We can jut make them strings and avoid worrying about
> > the
> > > > compatibility issues.
> > > >
> > >
> 

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Ron Dagostino]

+1 (non-binding)

Ron

On Tue, Jan 21, 2020 at 11:29 AM Manikumar  wrote:
>
> +1 (binding).
>
> Thanks for the KIP.
>
>
> On Tue, Jan 21, 2020 at 9:56 PM Ted Yu  wrote:
>
> > +1
> >
> > On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks for the KIP!
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков 
> > > wrote:
> > >
> > > > Hello.
> > > >
> > > > I would like to start vote for KIP-553: Disable all SSL protocols
> > except
> > > > TLSV1.2 by default.
> > > >
> > > > KIP -
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> > > > Discussion thread -
> > > >
> > >
> > https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
> > >
> >

Re: [DISCUSS] : KIP-562: Allow fetching a key from a single partition rather than iterating over all the stores on an instance

[Vinoth Chandar]

Chiming in a bit late here..

+1 This is a very valid improvement. Avoiding doing gets on irrelevant
partitions will improve performance and efficiency for IQs.

As an incremental improvement to the current APIs,  adding an option to
filter out based on partitions makes sense







On Mon, Jan 20, 2020 at 3:13 AM Navinder Brar
 wrote:

> Thanks John. If there are no other comments to be addressed, I will start
> a vote today so that we are on track for this release.~Navinder
>
>
> On Monday, January 20, 2020, 8:32 AM, John Roesler 
> wrote:
>
> Thanks, Navinder,
>
> The Param object looks a bit different than I would have done, but it
> certainly is explicit. We might have to deprecate those particular factory
> methods and move to a builder pattern if we need to add any more options in
> the future, but I’m fine with that possibility.
>
> The KIP also discusses some implementation details that aren’t necessary
> here. We really only need to see the public interfaces. We can discuss the
> implementation in the PR.
>
> That said, the public API part of the current proposal looks good to me! I
> would be a +1 if you called for a vote.
>
> Thanks,
> John
>
> On Sun, Jan 19, 2020, at 20:50, Navinder Brar wrote:
> > I have made some edits in the KIP, please take another look. It would
> > be great if we can push it in 2.5.0.
> > ~Navinder
> >
> >
> > On Sunday, January 19, 2020, 12:59 AM, Navinder Brar
> >  wrote:
> >
> > Sure John, I will update the StoreQueryParams with static factory
> > methods.
> > @Ted, we would need to create taskId only in case a user provides one
> > single partition. In case user wants to query all partitions of an
> > instance the current code is good enough where we iterate over all
> > stream threads and go over all taskIds to match the store. But in case
> > a user requests for a single partition-based store, we need to create a
> > taskId out of that partition and store name(using
> > internalTopologyBuilder class) and match with the taskIds belonging to
> > that instance. I will add the code in the KIP.
> >
> > On Sunday, 19 January, 2020, 12:47:08 am IST, Ted Yu
> >  wrote:
> >
> >  Looking at the current KIP-562:
> >
> > bq. Create a taskId from the combination of store name and partition
> > provided by the user
> >
> > I wonder if a single taskId would be used for the “all partitions” case.
> > If so, we need to choose a numerical value for the partition portion of
> the
> > taskId.
> >
> > On Sat, Jan 18, 2020 at 10:27 AM John Roesler 
> wrote:
> >
> > > Thanks, Ted!
> > >
> > > This makes sense, but it seems like we should lean towards explicit
> > > semantics in the public API. ‘-1’ meaning “all partitions” is
> reasonable,
> > > but not explicit. That’s why I suggested the Boolean for “all
> partitions”.
> > > I guess this also means getPartition() should either throw an
> exception or
> > > return null if the partition is unspecified.
> > >
> > > Thanks,
> > > John
> > >
> > > On Sat, Jan 18, 2020, at 08:43, Ted Yu wrote:
> > > > I wonder if the following two methods can be combined:
> > > >
> > > > Integer getPartition() // would be null if unset or if "all
> partitions"
> > > > boolean getAllLocalPartitions() // true/false if "all partitions"
> > > requested
> > > >
> > > > into:
> > > >
> > > > Integer getPartition() // would be null if unset or -1 if "all
> > > partitions"
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Jan 17, 2020 at 9:56 PM John Roesler 
> > > wrote:
> > > >
> > > > > Thanks, Navinder!
> > > > >
> > > > > I took a look at the KIP.
> > > > >
> > > > > We tend to use static factory methods instead of public
> constructors,
> > > and
> > > > > also builders for optional parameters.
> > > > >
> > > > > Given that, I think it would be more typical to have a factory
> method:
> > > > > storeQueryParams()
> > > > >
> > > > > and also builders for setting the optional parameters, like:
> > > > > withPartitions(List partitions)
> > > > > withStaleStoresEnabled()
> > > > > withStaleStoresDisabled()
> > > > >
> > > > >
> > > > > I was also thinking this over today, and it really seems like
> there are
> > > > > two main cases for specifying partitions,
> > > > > 1. you know exactly what partition you want. In this case, you'll
> only
> > > > > pass in a single number.
> > > > > 2. you want to get a handle on all the stores for this instance
> (the
> > > > > current behavior). In this case, it's not clear how to use
> > > withPartitions
> > > > > to achieve the goal, unless you want to apply a-priori knowledge
> of the
> > > > > number of partitions in the store. We could consider an empty
> list, or
> > > a
> > > > > null, to indicate "all", but that seems a little complicated.
> > > > >
> > > > > Thus, maybe it would actually be better to eschew withPartitions
> for
> > > now
> > > > > and instead just offer:
> > > > > withPartition(int partition)
> > > > > withAllLocalPartitions()
> > > > >
> > > > > and the getters:
> > > > > Integer getPartition() 

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Guozhang Wang]

Got it.

I was proposing that we do the "delayed async batch" but I think your
argument for complexity and pushing it out of the scope is convincing, so
instead I propose we do the synchronous mini batching still but obviously
it is already there :)  I'm +1 on the current proposal scope.

Guozhang

On Tue, Jan 21, 2020 at 10:16 AM Brian Byrne  wrote:

> Hi Guozhang,
>
> Ah, sorry, I misunderstood. Actually, this is solved for us today. How the
> producer works is that it maintains at most one inflight metadata fetch
> request at any time, where each request is tagged with the current
> (monotonically increasing) request version. This version is bumped whenever
> a new topic is encountered, and metadata fetching will continue to process
> while the latest metadata response's version is below the current version.
>
> So if a metadata request is in flight, and a number of threads produce to
> new topics, they'll be added to the working set but the next metadata
> request won't take place until the outstanding one returns. So their
> updates will be batched together. As you suggest, we can have a simple list
> that tracks unknown topics to isolate new vs. old topics.
>
> Thanks,
> Brian
>
>
>
> On Tue, Jan 21, 2020 at 10:04 AM Guozhang Wang  wrote:
>
> > Hi Brian,
> >
> > I think I buy the complexity and extra end-to-end-latency argument :) I'm
> > fine with delaying the asynchronous tech fetching to future works and
> keep
> > the current KIP's scope as-is for now. Under that case can we consider
> just
> > a minor implementation detail (since it is not affecting public APIs we
> > probably do not even need to list it, but just thinking loud here):
> >
> > In your proposal when we request for a topic of unknown metadata, we are
> > going to directly set the topic name as that singleton in the request.
> I'm
> > wondering for the scenario that KAFKA-8904 described, if the
> producer#send
> > for thousands of new topics are triggered sequentially by a single thread
> > or concurrent threads? If it's the latter, and we expect in such
> scenarios
> > we may have multiple topics being requests within a very short time, then
> > we can probably do sth. like this internally in a synchronized manner:
> >
> > 1) put the topic name into a list, as "unknown topics", then
> > 2) exhaust the list, and put all topics from that list to the request; if
> > the list is empty, it means it has been emptied by another thread so we
> > skip sending a new request and just wait for the returned metadata
> refresh.
> >
> > In most cases the list would just be a singleton with the one that thread
> > has just enqueued, but under extreme scenarios it can help batching a few
> > topic names probably (of course, I'm thinking about very extreme cases
> > here, assuming that's was what we've seen in 8904). Since these two steps
> > are very light-weighted, doing that in a synchronized block would not
> hurt
> > the concurrency too much.
> >
> >
> > Guozhang
> >
> >
> > On Tue, Jan 21, 2020 at 9:39 AM Brian Byrne  wrote:
> >
> > > Hi Guozhang,
> > >
> > > Your understanding of the rationale is accurate, and what you suggest
> is
> > > completely plausible, however I have a slightly different take on the
> > > situation.
> > >
> > > When the KIP was originally drafted, making KafkaProducer#send
> > asynchronous
> > > was one element of the changes (this is a little more general than (a),
> > but
> > > has similar implications). As you're aware, doing so would allow new
> > topics
> > > to aggregate since the producer could continue to push new records,
> > whereas
> > > today the producer thread is blocked waiting for resolution.
> > >
> > > However, there were concerns about changing client behavior
> unexpectedly
> > in
> > > this manner, and the change isn't as trivial as one would hope. For
> > > example, we'd have to introduce an intermediate queue of records for
> > topics
> > > without metadata, and have that play well with the buffer pool which
> > > ensures the memory limit isn't exceeded. A side effect is that a
> producer
> > > could hit 'memory full' conditions easier, which could have unintended
> > > consequences if, say, the model was setup such that different producer
> > > threads produced to a disjoint set of topics. Where one producer thread
> > was
> > > blocked waiting for new metadata, it could now push enough data to
> block
> > > all producer threads due to memory limits, so we'd need to be careful
> > here.
> > >
> > > For case (a) described, another concern would be adding additional a
> new
> > > source of latency (possibly seconds) for new topics. Not a huge issue,
> > but
> > > it is new behavior to existing clients and adds to the complexity of
> > > verifying no major regressions.
> > >
> > > It also wouldn't resolve all cases we're interested in. One behavior
> > we're
> > > witnessing is the following: a producer generates to a very large
> number
> > of
> > > topics (several thousand), however the period 

Build failed in Jenkins: kafka-trunk-jdk11 #1098

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update AclCommand help message to match implementation (#7990)


--
[...truncated 5.75 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

Re: [VOTE] KIP-558: Add Connect REST API endpoints to view the topics used by connectors in Kafka Connect

[Almog Gavra]

Another thanks from me! +1 (non-binding)

On Tue, Jan 21, 2020 at 11:04 AM Randall Hauch  wrote:

> Thanks again for the KIP and this improvement for Connect.
>
> +1 (binding)
>
> Randall
>
> On Tue, Jan 21, 2020 at 10:45 AM Tom Bentley  wrote:
>
> > +1 (non-binding). Thanks for the KIP Konstantine.
> >
> > On Sat, Jan 18, 2020 at 2:18 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to open the vote on KIP-558 that had a constructive flurry of
> > > discussions in the past few days, in order to give this KIP the
> > opportunity
> > > to be voted on by the current KIP deadline (Wed, Jan 22, 2020), if - of
> > > course - there's agreement upon its final form.
> > >
> > > KIP link here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
> > >
> > > Best regards,
> > > Konstantine
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk8 #4174

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update AclCommand help message to match implementation (#7990)


--
[...truncated 5.72 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > 

Re: [VOTE] KIP-558: Add Connect REST API endpoints to view the topics used by connectors in Kafka Connect

[Randall Hauch]

Thanks again for the KIP and this improvement for Connect.

+1 (binding)

Randall

On Tue, Jan 21, 2020 at 10:45 AM Tom Bentley  wrote:

> +1 (non-binding). Thanks for the KIP Konstantine.
>
> On Sat, Jan 18, 2020 at 2:18 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Hi all,
> >
> > I'd like to open the vote on KIP-558 that had a constructive flurry of
> > discussions in the past few days, in order to give this KIP the
> opportunity
> > to be voted on by the current KIP deadline (Wed, Jan 22, 2020), if - of
> > course - there's agreement upon its final form.
> >
> > KIP link here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
> >
> > Best regards,
> > Konstantine
> >
>

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Konstantine Karantasis]

Randall, I updated the KIP to add your suggestion above and better explain
the few points that were outstanding since last week.

Almog, I agree that 'timestamp' was too generic. I'm ok with
'discoverTimestamp'. I updated the KIP with your naming suggestion.

Best,
Konstantine


On Tue, Jan 21, 2020 at 10:43 AM Almog Gavra  wrote:

> Thanks again Konstantine - really excited about this KIP!
>
> I'm about ready to +1 (non-binding) it with just one comment left: What do
> you think about changing the timestamp field to "discoverTimestamp" or
> something like that to indicate that it is the timestamp of the _first_
> time we recorded/discovered this topic. This is important if we later want
> to add a "mostRecentTimestamp" field in the future to distinguish between
> the two.
>
> Almog
>
> On Tue, Jan 21, 2020 at 8:01 AM Randall Hauch  wrote:
>
> > Thanks, Konstantine.
> >
> > One minor request to clarify the following sentence:
> >
> >
> > As soon as a worker detects the addition of a topic to a connector's set
> of
> > active topics, the worker will cease to post update messages to the
> > status.storage.topic for that connector.
> >
> >
> > As it stands, it sounds like the worker will not write *any more active
> > topic records for this or any connectors* to the topic specified by the
> > `status.storage.topic` worker configuration once the worker detects (by
> > reading) a new active topic. I suspect that this is not the intention,
> and
> > that instead it is trying to say that no more messages *for this topic
> and
> > connector*. IOW, something more like:
> >
> >
> > As soon as a worker detects the addition of a topic to a connector's set
> of
> > active topics, the worker will not post to the status.storage.topic
> > additional update records for the connector and this newly-detected
> active
> > topic.
> >
> >
> > Otherwise, this KIP looks great!
> >
> > Best regards,
> >
> > Randall
> >
> > On Fri, Jan 17, 2020 at 8:04 PM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Hi all,
> > >
> > > I've updated KIP-558 with the following based on our previous
> discussion:
> > >
> > > * Added timestamp to the metadata (the record value).
> > > * The KIP now mentions a metric-based implementation in the Rejected
> > > Alternatives section.
> > > * The record key format is now using the single character ':' as a
> > > separator between topic-${topic name} and connector-${connector name}
> > > * Added a bullet point to mention that the topic storing the new
> > > information can be a partitioned topic.
> > > * The KIP mentions that the feature does not require rebuilding
> > connectors
> > > (no changes in public interfaces/classes).
> > > * Added a security section.
> > > * KIP preserves symmetry with respect to reset between both types of
> > > connectors and keeps reset and config as separate, unrelated endpoints.
> > >
> > > Given than we made significant progress these past few days and only a
> > few
> > > minor improvements in the KIPs text are remaining, I'd like to start
> the
> > > vote today, so that we give this KIP the necessary time (72 hours) to
> > have
> > > a chance to be voted by the KIP deadline next Wednesday, Jan 22nd.
> > > Let's return here, or the main vote thread for any comments (either
> minor
> > > to major).
> > >
> > > Best,
> > > Konstantine
> > >
> > >
> > >
> > > On Fri, Jan 17, 2020 at 3:51 PM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > >
> > > > Thanks for the follow up Chris. Replies below:
> > > >
> > > > On Fri, Jan 17, 2020 at 3:07 PM Christopher Egerton <
> > chr...@confluent.io
> > > >
> > > > wrote:
> > > >
> > > >> Thanks, Konstantine. Just a few more questions:
> > > >>
> > > >> > > 2. What is the motivation for the `topic.tracking.allow.reset`
> > > config?
> > > >> Is
> > > >> > > there any anticipated case where it would be useful to have
> topic
> > > >> tracking
> > > >> > > enabled but with resets disabled? We could easily add this
> > > >> configuration
> > > >> > > later if a use case arises, but if we add it now it'll be
> > difficult
> > > to
> > > >> > > remove.
> > > >> > >
> > > >>
> > > >> > The motivation is for operators of a Connect cluster to be able to
> > > >> disable
> > > >> > resetting the history of active topics altogether, while allowing
> at
> > > the
> > > >> > same time to view the active sets.
> > > >>
> > > >> What I was trying to ask was, is there a use case for enabling the
> > > latter
> > > >> but not the former? We should be careful about adding extra worker
> > > configs
> > > >> and unless we can anticipate a reasonable scenario in which this
> would
> > > >> happen, we should err on the side of caution and avoid adding a
> config
> > > >> that
> > > >> would be difficult to remove later but, comparably, much easier to
> > add.
> > > >>
> > > >
> > > > The application use case is the ability to have immutable histories
> of
> > > > topic usage or control when resets are 

Re: [VOTE] KIP-562: Allow fetching a key from a single partition rather than iterating over all the stores on an instance

[Navinder Brar]

Thanks, Guozhang. I agree it makes total sense. I will make the edits.~Navinder 
 

On Tuesday, 21 January, 2020, 11:00:32 pm IST, Guozhang Wang 
 wrote:  
 
 Hello Navinder,

Thanks for brining up this proposal. I made a quick pass on that and
overall I think I agree with your ideas. Just a few thoughts about the
public APIs:

1) As we are adding a new overload to `KafkaStreams#store`, could we just
add the storeName and queryableStoreType as part of StoreQueryParam, and
leaving that the only parameter of the function?

2) along with 1), for the static constructors, instead of iterating over
all possible combos I'd suggest we make constructors with only, say,
storeName, and then adding `withXXX()` setters to set other fields. This is
in case we want to add more param fields into the object, that we do not
need to exponentially adding and deprecating the static constructors.


Guozhang


On Mon, Jan 20, 2020 at 10:42 AM Navinder Brar
 wrote:

> Hello all,
>
> I'd like to propose a vote to serve keys from a specific partition-store
> instead of iterating over all the local stores of an instance to locate the
> key, as which happens currently.
> The full KIP is provided here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-562%3A+Allow+fetching+a+key+from+a+single+partition+rather+than+iterating+over+all+the+stores+on+an+instance
>
>
> Thanks,
> Navinder
>


-- 
-- Guozhang
  

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Almog Gavra]

Thanks again Konstantine - really excited about this KIP!

I'm about ready to +1 (non-binding) it with just one comment left: What do
you think about changing the timestamp field to "discoverTimestamp" or
something like that to indicate that it is the timestamp of the _first_
time we recorded/discovered this topic. This is important if we later want
to add a "mostRecentTimestamp" field in the future to distinguish between
the two.

Almog

On Tue, Jan 21, 2020 at 8:01 AM Randall Hauch  wrote:

> Thanks, Konstantine.
>
> One minor request to clarify the following sentence:
>
>
> As soon as a worker detects the addition of a topic to a connector's set of
> active topics, the worker will cease to post update messages to the
> status.storage.topic for that connector.
>
>
> As it stands, it sounds like the worker will not write *any more active
> topic records for this or any connectors* to the topic specified by the
> `status.storage.topic` worker configuration once the worker detects (by
> reading) a new active topic. I suspect that this is not the intention, and
> that instead it is trying to say that no more messages *for this topic and
> connector*. IOW, something more like:
>
>
> As soon as a worker detects the addition of a topic to a connector's set of
> active topics, the worker will not post to the status.storage.topic
> additional update records for the connector and this newly-detected active
> topic.
>
>
> Otherwise, this KIP looks great!
>
> Best regards,
>
> Randall
>
> On Fri, Jan 17, 2020 at 8:04 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Hi all,
> >
> > I've updated KIP-558 with the following based on our previous discussion:
> >
> > * Added timestamp to the metadata (the record value).
> > * The KIP now mentions a metric-based implementation in the Rejected
> > Alternatives section.
> > * The record key format is now using the single character ':' as a
> > separator between topic-${topic name} and connector-${connector name}
> > * Added a bullet point to mention that the topic storing the new
> > information can be a partitioned topic.
> > * The KIP mentions that the feature does not require rebuilding
> connectors
> > (no changes in public interfaces/classes).
> > * Added a security section.
> > * KIP preserves symmetry with respect to reset between both types of
> > connectors and keeps reset and config as separate, unrelated endpoints.
> >
> > Given than we made significant progress these past few days and only a
> few
> > minor improvements in the KIPs text are remaining, I'd like to start the
> > vote today, so that we give this KIP the necessary time (72 hours) to
> have
> > a chance to be voted by the KIP deadline next Wednesday, Jan 22nd.
> > Let's return here, or the main vote thread for any comments (either minor
> > to major).
> >
> > Best,
> > Konstantine
> >
> >
> >
> > On Fri, Jan 17, 2020 at 3:51 PM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > >
> > > Thanks for the follow up Chris. Replies below:
> > >
> > > On Fri, Jan 17, 2020 at 3:07 PM Christopher Egerton <
> chr...@confluent.io
> > >
> > > wrote:
> > >
> > >> Thanks, Konstantine. Just a few more questions:
> > >>
> > >> > > 2. What is the motivation for the `topic.tracking.allow.reset`
> > config?
> > >> Is
> > >> > > there any anticipated case where it would be useful to have topic
> > >> tracking
> > >> > > enabled but with resets disabled? We could easily add this
> > >> configuration
> > >> > > later if a use case arises, but if we add it now it'll be
> difficult
> > to
> > >> > > remove.
> > >> > >
> > >>
> > >> > The motivation is for operators of a Connect cluster to be able to
> > >> disable
> > >> > resetting the history of active topics altogether, while allowing at
> > the
> > >> > same time to view the active sets.
> > >>
> > >> What I was trying to ask was, is there a use case for enabling the
> > latter
> > >> but not the former? We should be careful about adding extra worker
> > configs
> > >> and unless we can anticipate a reasonable scenario in which this would
> > >> happen, we should err on the side of caution and avoid adding a config
> > >> that
> > >> would be difficult to remove later but, comparably, much easier to
> add.
> > >>
> > >
> > > The application use case is the ability to have immutable histories of
> > > topic usage or control when resets are allowed and how they are
> performed
> > > (e.g. resets could be allowed briefly during a maintenance phase and
> get
> > > disabled again).
> > > I'm also never thrilled when I add an extra configuration parameter.
> > > However namespacing here will help with the extra cognitive burden.
> > > Similarly the defaults should cover most use cases too.
> > >
> > > > > 5. As far as automatic resets for sink connectors go, I agree with
> > your
> > >> > > reasoning about the inherent asymmetry between sinks and sources,
> > and
> > >> with
> > >> > > the motivation to avoid confusing users by listing
> 

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Brian Byrne]

Hi Guozhang,

Ah, sorry, I misunderstood. Actually, this is solved for us today. How the
producer works is that it maintains at most one inflight metadata fetch
request at any time, where each request is tagged with the current
(monotonically increasing) request version. This version is bumped whenever
a new topic is encountered, and metadata fetching will continue to process
while the latest metadata response's version is below the current version.

So if a metadata request is in flight, and a number of threads produce to
new topics, they'll be added to the working set but the next metadata
request won't take place until the outstanding one returns. So their
updates will be batched together. As you suggest, we can have a simple list
that tracks unknown topics to isolate new vs. old topics.

Thanks,
Brian



On Tue, Jan 21, 2020 at 10:04 AM Guozhang Wang  wrote:

> Hi Brian,
>
> I think I buy the complexity and extra end-to-end-latency argument :) I'm
> fine with delaying the asynchronous tech fetching to future works and keep
> the current KIP's scope as-is for now. Under that case can we consider just
> a minor implementation detail (since it is not affecting public APIs we
> probably do not even need to list it, but just thinking loud here):
>
> In your proposal when we request for a topic of unknown metadata, we are
> going to directly set the topic name as that singleton in the request. I'm
> wondering for the scenario that KAFKA-8904 described, if the producer#send
> for thousands of new topics are triggered sequentially by a single thread
> or concurrent threads? If it's the latter, and we expect in such scenarios
> we may have multiple topics being requests within a very short time, then
> we can probably do sth. like this internally in a synchronized manner:
>
> 1) put the topic name into a list, as "unknown topics", then
> 2) exhaust the list, and put all topics from that list to the request; if
> the list is empty, it means it has been emptied by another thread so we
> skip sending a new request and just wait for the returned metadata refresh.
>
> In most cases the list would just be a singleton with the one that thread
> has just enqueued, but under extreme scenarios it can help batching a few
> topic names probably (of course, I'm thinking about very extreme cases
> here, assuming that's was what we've seen in 8904). Since these two steps
> are very light-weighted, doing that in a synchronized block would not hurt
> the concurrency too much.
>
>
> Guozhang
>
>
> On Tue, Jan 21, 2020 at 9:39 AM Brian Byrne  wrote:
>
> > Hi Guozhang,
> >
> > Your understanding of the rationale is accurate, and what you suggest is
> > completely plausible, however I have a slightly different take on the
> > situation.
> >
> > When the KIP was originally drafted, making KafkaProducer#send
> asynchronous
> > was one element of the changes (this is a little more general than (a),
> but
> > has similar implications). As you're aware, doing so would allow new
> topics
> > to aggregate since the producer could continue to push new records,
> whereas
> > today the producer thread is blocked waiting for resolution.
> >
> > However, there were concerns about changing client behavior unexpectedly
> in
> > this manner, and the change isn't as trivial as one would hope. For
> > example, we'd have to introduce an intermediate queue of records for
> topics
> > without metadata, and have that play well with the buffer pool which
> > ensures the memory limit isn't exceeded. A side effect is that a producer
> > could hit 'memory full' conditions easier, which could have unintended
> > consequences if, say, the model was setup such that different producer
> > threads produced to a disjoint set of topics. Where one producer thread
> was
> > blocked waiting for new metadata, it could now push enough data to block
> > all producer threads due to memory limits, so we'd need to be careful
> here.
> >
> > For case (a) described, another concern would be adding additional a new
> > source of latency (possibly seconds) for new topics. Not a huge issue,
> but
> > it is new behavior to existing clients and adds to the complexity of
> > verifying no major regressions.
> >
> > It also wouldn't resolve all cases we're interested in. One behavior
> we're
> > witnessing is the following: a producer generates to a very large number
> of
> > topics (several thousand), however the period of consecutive records for
> a
> > topic can often be beyond the current hard-coded expiry of 5 minutes.
> > Therefore, when the producer does submit a request for this topic after 5
> > minutes, it has to request all of the topic metadata again. While
> batching
> > new topics in the start-up case would definitely help, here it would
> likely
> > be lost effort.
> >
> > Being able to increase the metadata eviction for the above case would
> > improve things, but there's no perfect value when consecutive produce
> times
> > can be modelled as a probability 

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Guozhang Wang]

Hi Brian,

I think I buy the complexity and extra end-to-end-latency argument :) I'm
fine with delaying the asynchronous tech fetching to future works and keep
the current KIP's scope as-is for now. Under that case can we consider just
a minor implementation detail (since it is not affecting public APIs we
probably do not even need to list it, but just thinking loud here):

In your proposal when we request for a topic of unknown metadata, we are
going to directly set the topic name as that singleton in the request. I'm
wondering for the scenario that KAFKA-8904 described, if the producer#send
for thousands of new topics are triggered sequentially by a single thread
or concurrent threads? If it's the latter, and we expect in such scenarios
we may have multiple topics being requests within a very short time, then
we can probably do sth. like this internally in a synchronized manner:

1) put the topic name into a list, as "unknown topics", then
2) exhaust the list, and put all topics from that list to the request; if
the list is empty, it means it has been emptied by another thread so we
skip sending a new request and just wait for the returned metadata refresh.

In most cases the list would just be a singleton with the one that thread
has just enqueued, but under extreme scenarios it can help batching a few
topic names probably (of course, I'm thinking about very extreme cases
here, assuming that's was what we've seen in 8904). Since these two steps
are very light-weighted, doing that in a synchronized block would not hurt
the concurrency too much.


Guozhang


On Tue, Jan 21, 2020 at 9:39 AM Brian Byrne  wrote:

> Hi Guozhang,
>
> Your understanding of the rationale is accurate, and what you suggest is
> completely plausible, however I have a slightly different take on the
> situation.
>
> When the KIP was originally drafted, making KafkaProducer#send asynchronous
> was one element of the changes (this is a little more general than (a), but
> has similar implications). As you're aware, doing so would allow new topics
> to aggregate since the producer could continue to push new records, whereas
> today the producer thread is blocked waiting for resolution.
>
> However, there were concerns about changing client behavior unexpectedly in
> this manner, and the change isn't as trivial as one would hope. For
> example, we'd have to introduce an intermediate queue of records for topics
> without metadata, and have that play well with the buffer pool which
> ensures the memory limit isn't exceeded. A side effect is that a producer
> could hit 'memory full' conditions easier, which could have unintended
> consequences if, say, the model was setup such that different producer
> threads produced to a disjoint set of topics. Where one producer thread was
> blocked waiting for new metadata, it could now push enough data to block
> all producer threads due to memory limits, so we'd need to be careful here.
>
> For case (a) described, another concern would be adding additional a new
> source of latency (possibly seconds) for new topics. Not a huge issue, but
> it is new behavior to existing clients and adds to the complexity of
> verifying no major regressions.
>
> It also wouldn't resolve all cases we're interested in. One behavior we're
> witnessing is the following: a producer generates to a very large number of
> topics (several thousand), however the period of consecutive records for a
> topic can often be beyond the current hard-coded expiry of 5 minutes.
> Therefore, when the producer does submit a request for this topic after 5
> minutes, it has to request all of the topic metadata again. While batching
> new topics in the start-up case would definitely help, here it would likely
> be lost effort.
>
> Being able to increase the metadata eviction for the above case would
> improve things, but there's no perfect value when consecutive produce times
> can be modelled as a probability distribution. Rather, the better solution
> could be what we discussed earlier, where we'd lower the eviction timeout,
> but make the cost of fetching uncached topic metadata much smaller.
>
> The changes in the KIP were meant to improve the general case without
> affecting external client behavior, and then the plan was to fix the
> asynchronous send in the next iteration, if necessary. Point (b) is along
> the lines of the latest revision: only send requests for uncached topics,
> if they exist, otherwise request the full working set. Piggy-backing was
> originally included, but removed because its utility was in doubt.
>
> So to summarize, you're correct in that asynchronous topic fetching would
> be a big improvement, and should be an item of future work. However what
> this KIP proposes should be the safest/easiest set of changes to resolve
> the current pain points. Please let me know if you agree/disagree with this
> assessment.
>
> Thanks,
> Brian
>
>
> On Mon, Jan 20, 2020 at 10:52 AM Guozhang Wang  wrote:
>
> > Hello Brian,
> 

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Brian Byrne]

Hi Guozhang,

Your understanding of the rationale is accurate, and what you suggest is
completely plausible, however I have a slightly different take on the
situation.

When the KIP was originally drafted, making KafkaProducer#send asynchronous
was one element of the changes (this is a little more general than (a), but
has similar implications). As you're aware, doing so would allow new topics
to aggregate since the producer could continue to push new records, whereas
today the producer thread is blocked waiting for resolution.

However, there were concerns about changing client behavior unexpectedly in
this manner, and the change isn't as trivial as one would hope. For
example, we'd have to introduce an intermediate queue of records for topics
without metadata, and have that play well with the buffer pool which
ensures the memory limit isn't exceeded. A side effect is that a producer
could hit 'memory full' conditions easier, which could have unintended
consequences if, say, the model was setup such that different producer
threads produced to a disjoint set of topics. Where one producer thread was
blocked waiting for new metadata, it could now push enough data to block
all producer threads due to memory limits, so we'd need to be careful here.

For case (a) described, another concern would be adding additional a new
source of latency (possibly seconds) for new topics. Not a huge issue, but
it is new behavior to existing clients and adds to the complexity of
verifying no major regressions.

It also wouldn't resolve all cases we're interested in. One behavior we're
witnessing is the following: a producer generates to a very large number of
topics (several thousand), however the period of consecutive records for a
topic can often be beyond the current hard-coded expiry of 5 minutes.
Therefore, when the producer does submit a request for this topic after 5
minutes, it has to request all of the topic metadata again. While batching
new topics in the start-up case would definitely help, here it would likely
be lost effort.

Being able to increase the metadata eviction for the above case would
improve things, but there's no perfect value when consecutive produce times
can be modelled as a probability distribution. Rather, the better solution
could be what we discussed earlier, where we'd lower the eviction timeout,
but make the cost of fetching uncached topic metadata much smaller.

The changes in the KIP were meant to improve the general case without
affecting external client behavior, and then the plan was to fix the
asynchronous send in the next iteration, if necessary. Point (b) is along
the lines of the latest revision: only send requests for uncached topics,
if they exist, otherwise request the full working set. Piggy-backing was
originally included, but removed because its utility was in doubt.

So to summarize, you're correct in that asynchronous topic fetching would
be a big improvement, and should be an item of future work. However what
this KIP proposes should be the safest/easiest set of changes to resolve
the current pain points. Please let me know if you agree/disagree with this
assessment.

Thanks,
Brian


On Mon, Jan 20, 2020 at 10:52 AM Guozhang Wang  wrote:

> Hello Brian,
>
> I looked at the new proposal again, and I'd like to reason about its
> rationale from the listed motivations in your wiki:
>
> 1) more RPCs: we may send metadata requests more frequently than
> appropriate. This is especially the case during producer start-up, where
> the more topics it needs to send to, the more metadata requests it needs to
> send. This the original reported issue as in KAFKA-8904.
>
> 2) large RPCs: we including all topics in the work set when sending
> metadata request. But I think our conjecture (as Colin has pointed out) is
> that this alone is fine most of the time, assuming e.g. you are sending
> such large RPC only once every 10 minutes. It is only because of 1) where
> you are sending large RPC too frequently which is a common issue.
>
> 3) we want to have a configurable eviction period than hard-coded values. I
> consider it as a semi-orthogonal motivation compared with 2) / 3) but we
> wanted to piggy-back this fix along with the KIP.
>
> So from there, 1) and 2) does not contradict to each other since our belief
> is that large RPCs is usually okay as long as it is not large-and-frequent
> RPCs, and we actually prefer large-infrequent RPC > smaller-frequent RPC >
> large-and-frequent RPC (of course).
>
> The current proposal basically tries to un-tangle 2) from 1), i.e. for the
> scenario of KAFKA-8904 it would result in smaller-frequent RPC during
> startup than large-and-frequent RPC. But I'm wondering why don't we just do
> even better and make it large-infrequent RPC? More specifically, just like
> Lucas suggested in the ticket:
>
> a. when there's new topic with unknown metadata enqueued, instead of
> requesting a metadata immediately just delay it for a short period (no more
> than 

[VOTE] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[David Jacot]

Hi all,

I would like to start a vote on KIP-559: Make the Kafka Protocol Friendlier
with L7 Proxies.

The KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies

Thanks,
David

Re: [VOTE] KIP-562: Allow fetching a key from a single partition rather than iterating over all the stores on an instance

[Guozhang Wang]

Hello Navinder,

Thanks for brining up this proposal. I made a quick pass on that and
overall I think I agree with your ideas. Just a few thoughts about the
public APIs:

1) As we are adding a new overload to `KafkaStreams#store`, could we just
add the storeName and queryableStoreType as part of StoreQueryParam, and
leaving that the only parameter of the function?

2) along with 1), for the static constructors, instead of iterating over
all possible combos I'd suggest we make constructors with only, say,
storeName, and then adding `withXXX()` setters to set other fields. This is
in case we want to add more param fields into the object, that we do not
need to exponentially adding and deprecating the static constructors.


Guozhang


On Mon, Jan 20, 2020 at 10:42 AM Navinder Brar
 wrote:

> Hello all,
>
> I'd like to propose a vote to serve keys from a specific partition-store
> instead of iterating over all the local stores of an instance to locate the
> key, as which happens currently.
> The full KIP is provided here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-562%3A+Allow+fetching+a+key+from+a+single+partition+rather+than+iterating+over+all+the+stores+on+an+instance
>
>
> Thanks,
> Navinder
>


-- 
-- Guozhang

Re: Create KIP permission.

[Bill Bejeck]

Done.

Thanks for your interest in Apache Kafka.

-Bill

On Tue, Jan 21, 2020 at 11:35 AM Игорь Мартемьянов 
wrote:

> Username: ledostuff
>
> вт, 21 янв. 2020 г., 15:43 Игорь Мартемьянов :
>
> > Hello there.
> > I'd like to request permission to Kafka confluence page to be able to
> > create KIPs.
> >
>

Re: Is there a way to auto scale topic partitions in kafka?

[Adam Bellemare]

There is no way to automatically scale it, but you could write a script to
increase the partition count using the command line tools, and trigger it
on certain metrics.

One thing to consider is that any *keyed* events would need to be rewritten
to topics that have their partition count increased. This is to ensure that
keyed data locality is preserved within each partition, such that all
events of a single key stay in a single partition. If you don't care about
data-locality, then you can increase the partition count without concern.





On Tue, Jan 21, 2020 at 11:35 AM Pushkar Deole  wrote:

> Hello Dev community,
>
> Got no response from user community on below query. Can you respond back on
> this please?
>
> -- Forwarded message -
> From: Pushkar Deole 
> Date: Fri, Jan 17, 2020 at 1:46 PM
> Subject: Is there a way to auto scale topic partitions in kafka?
> To: 
>
>
> Hello,
>
> I am working on developing a microservice based system which uses kafka as
> a messaging infrastructure. The microservices application are mainly kafka
> consumers and kafka streams applications and are deployed as docker
> containers on kubernetes.
>
> The system should be designed to be auto scalable for which we are using
> Horizontal Pod Autoscaler feature of kubernetes which allows to instantiate
> more number of pods if a certain metric (e.g. cpu utilization) touches the
> threshold or reduce the pods in case the metric is way below the threshold.
> However, the problem is number of partitions in kafka are fixed so even if
> load on the system increases and the number of consumer pods are
> autoscaled, it could not be scaled beyond the number of partitions.
> So, after a point where number of pods is equal to number of partitions,
> the system can't be scaled beyond that.
> Is there a way to autoscale number of partitions also in kafka so the
> system can be auto scaled in cloud?
>

Re: [VOTE] KIP-558: Add Connect REST API endpoints to view the topics used by connectors in Kafka Connect

[Tom Bentley]

+1 (non-binding). Thanks for the KIP Konstantine.

On Sat, Jan 18, 2020 at 2:18 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Hi all,
>
> I'd like to open the vote on KIP-558 that had a constructive flurry of
> discussions in the past few days, in order to give this KIP the opportunity
> to be voted on by the current KIP deadline (Wed, Jan 22, 2020), if - of
> course - there's agreement upon its final form.
>
> KIP link here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-558%3A+Track+the+set+of+actively+used+topics+by+connectors+in+Kafka+Connect
>
> Best regards,
> Konstantine
>

Re: Add to Dev mailing list

[Bruno Cadonna]

Hi Deepak,

Subscription to the mailing lists is self service. See
https://kafka.apache.org/contact

Best,
Bruno

On Tue, Jan 21, 2020 at 8:35 AM Deepak Goyal  wrote:
>
> Hi,
>
> Can you please add me to the dev mailing list
>
> --
> Deepak Goyal (+919502447100)

Create KIP permission.

[Игорь Мартемьянов]

Hello there.
I'd like to request permission to Kafka confluence page to be able to
create KIPs.

Re: Create KIP permission.

[Игорь Мартемьянов]

Username: ledostuff

вт, 21 янв. 2020 г., 15:43 Игорь Мартемьянов :

> Hello there.
> I'd like to request permission to Kafka confluence page to be able to
> create KIPs.
>

Add to Dev mailing list

[Deepak Goyal]

Hi,

Can you please add me to the dev mailing list

-- 
Deepak Goyal (+919502447100)

Fwd: Is there a way to auto scale topic partitions in kafka?

[Pushkar Deole]

Hello Dev community,

Got no response from user community on below query. Can you respond back on
this please?

-- Forwarded message -
From: Pushkar Deole 
Date: Fri, Jan 17, 2020 at 1:46 PM
Subject: Is there a way to auto scale topic partitions in kafka?
To: 


Hello,

I am working on developing a microservice based system which uses kafka as
a messaging infrastructure. The microservices application are mainly kafka
consumers and kafka streams applications and are deployed as docker
containers on kubernetes.

The system should be designed to be auto scalable for which we are using
Horizontal Pod Autoscaler feature of kubernetes which allows to instantiate
more number of pods if a certain metric (e.g. cpu utilization) touches the
threshold or reduce the pods in case the metric is way below the threshold.
However, the problem is number of partitions in kafka are fixed so even if
load on the system increases and the number of consumer pods are
autoscaled, it could not be scaled beyond the number of partitions.
So, after a point where number of pods is equal to number of partitions,
the system can't be scaled beyond that.
Is there a way to autoscale number of partitions also in kafka so the
system can be auto scaled in cloud?

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Manikumar]

+1 (binding).

Thanks for the KIP.


On Tue, Jan 21, 2020 at 9:56 PM Ted Yu  wrote:

> +1
>
> On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram 
> wrote:
>
> > +1 (binding)
> >
> > Thanks for the KIP!
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков 
> > wrote:
> >
> > > Hello.
> > >
> > > I would like to start vote for KIP-553: Disable all SSL protocols
> except
> > > TLSV1.2 by default.
> > >
> > > KIP -
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> > > Discussion thread -
> > >
> >
> https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
> >
>

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Ted Yu]

+1

On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Thanks for the KIP!
>
> Regards,
>
> Rajini
>
>
> On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков 
> wrote:
>
> > Hello.
> >
> > I would like to start vote for KIP-553: Disable all SSL protocols except
> > TLSV1.2 by default.
> >
> > KIP -
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> > Discussion thread -
> >
> https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
>

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Rajini Sivaram]

+1 (binding)

Thanks for the KIP!

Regards,

Rajini


On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков  wrote:

> Hello.
>
> I would like to start vote for KIP-553: Disable all SSL protocols except
> TLSV1.2 by default.
>
> KIP -
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> Discussion thread -
> https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Randall Hauch]

Thanks, Konstantine.

One minor request to clarify the following sentence:


As soon as a worker detects the addition of a topic to a connector's set of
active topics, the worker will cease to post update messages to the
status.storage.topic for that connector.


As it stands, it sounds like the worker will not write *any more active
topic records for this or any connectors* to the topic specified by the
`status.storage.topic` worker configuration once the worker detects (by
reading) a new active topic. I suspect that this is not the intention, and
that instead it is trying to say that no more messages *for this topic and
connector*. IOW, something more like:


As soon as a worker detects the addition of a topic to a connector's set of
active topics, the worker will not post to the status.storage.topic
additional update records for the connector and this newly-detected active
topic.


Otherwise, this KIP looks great!

Best regards,

Randall

On Fri, Jan 17, 2020 at 8:04 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Hi all,
>
> I've updated KIP-558 with the following based on our previous discussion:
>
> * Added timestamp to the metadata (the record value).
> * The KIP now mentions a metric-based implementation in the Rejected
> Alternatives section.
> * The record key format is now using the single character ':' as a
> separator between topic-${topic name} and connector-${connector name}
> * Added a bullet point to mention that the topic storing the new
> information can be a partitioned topic.
> * The KIP mentions that the feature does not require rebuilding connectors
> (no changes in public interfaces/classes).
> * Added a security section.
> * KIP preserves symmetry with respect to reset between both types of
> connectors and keeps reset and config as separate, unrelated endpoints.
>
> Given than we made significant progress these past few days and only a few
> minor improvements in the KIPs text are remaining, I'd like to start the
> vote today, so that we give this KIP the necessary time (72 hours) to have
> a chance to be voted by the KIP deadline next Wednesday, Jan 22nd.
> Let's return here, or the main vote thread for any comments (either minor
> to major).
>
> Best,
> Konstantine
>
>
>
> On Fri, Jan 17, 2020 at 3:51 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> >
> > Thanks for the follow up Chris. Replies below:
> >
> > On Fri, Jan 17, 2020 at 3:07 PM Christopher Egerton  >
> > wrote:
> >
> >> Thanks, Konstantine. Just a few more questions:
> >>
> >> > > 2. What is the motivation for the `topic.tracking.allow.reset`
> config?
> >> Is
> >> > > there any anticipated case where it would be useful to have topic
> >> tracking
> >> > > enabled but with resets disabled? We could easily add this
> >> configuration
> >> > > later if a use case arises, but if we add it now it'll be difficult
> to
> >> > > remove.
> >> > >
> >>
> >> > The motivation is for operators of a Connect cluster to be able to
> >> disable
> >> > resetting the history of active topics altogether, while allowing at
> the
> >> > same time to view the active sets.
> >>
> >> What I was trying to ask was, is there a use case for enabling the
> latter
> >> but not the former? We should be careful about adding extra worker
> configs
> >> and unless we can anticipate a reasonable scenario in which this would
> >> happen, we should err on the side of caution and avoid adding a config
> >> that
> >> would be difficult to remove later but, comparably, much easier to add.
> >>
> >
> > The application use case is the ability to have immutable histories of
> > topic usage or control when resets are allowed and how they are performed
> > (e.g. resets could be allowed briefly during a maintenance phase and get
> > disabled again).
> > I'm also never thrilled when I add an extra configuration parameter.
> > However namespacing here will help with the extra cognitive burden.
> > Similarly the defaults should cover most use cases too.
> >
> > > > 5. As far as automatic resets for sink connectors go, I agree with
> your
> >> > > reasoning about the inherent asymmetry between sinks and sources,
> and
> >> with
> >> > > the motivation to avoid confusing users by listing
> no-longer-consumed
> >> > > topics in the active topics for a sink connector. I think that this
> >> > > asymmetry is worth avoiding a scenario where a connector is
> >> reconfigured to
> >> > > only consume from topic "foo" but, from a prior configuration, topic
> >> "bar"
> >> > > is still listed in its active topics.
> >> > > I do want to request clarification on the meaning of the phrase "any
> >> topics
> >> > > no longer consumed" as used under the header "Restarting,
> >> reconfiguring
> >> or
> >> > > deleting a connector". Does this mean that the current set of active
> >> topics
> >> > > for the connector will be filtered and any that are longer contained
> >> in
> >> the
> >> > > sink connector's "topics" config or matched by its 

[VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Николай Ижиков]

Hello.

I would like to start vote for KIP-553: Disable all SSL protocols except 
TLSV1.2 by default.

KIP - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
Discussion thread - 
https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E

[jira] [Resolved] (KAFKA-9457) Flaky test org.apache.kafka.common.network.SelectorTest.testGracefulClose

[Rajini Sivaram (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-9457.
---
  Reviewer: Manikumar
Resolution: Fixed

> Flaky test org.apache.kafka.common.network.SelectorTest.testGracefulClose
> -
>
> Key: KAFKA-9457
> URL: https://issues.apache.org/jira/browse/KAFKA-9457
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.5.0
>
>
> org.apache.kafka.common.network.SelectorTest.testGracefulClose has been 
> failing a lot in PR builds:
> {{java.lang.AssertionError: expected:<1> but was:<0>}}
> {{ at org.junit.Assert.fail(Assert.java:89)}}
> {{ at org.junit.Assert.failNotEquals(Assert.java:835)}}
> {{ at org.junit.Assert.assertEquals(Assert.java:647)}}
> {{ at org.junit.Assert.assertEquals(Assert.java:633)}}
> {{ at 
> org.apache.kafka.common.network.SelectorTest.testGracefulClose(SelectorTest.java:588)}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Call for presentations for ApacheCon North America 2020 now open

[Rich Bowen]

Dear Apache enthusiast,

(You’re receiving this message because you are subscribed to one or more 
project mailing lists at the Apache Software Foundation.)


The call for presentations for ApacheCon North America 2020 is now open 
at https://apachecon.com/acna2020/cfp


ApacheCon will be held at the Sheraton, New Orleans, September 28th 
through October 2nd, 2020.


As in past years, ApacheCon will feature tracks focusing on the various 
technologies within the Apache ecosystem, and so the call for 
presentations will ask you to select one of those tracks, or “General” 
if the content falls outside of one of our already-organized tracks. 
These tracks are:


Karaf
Internet of Things
Fineract
Community
Content Delivery
Solr/Lucene (Search)
Gobblin/Big Data Integration
Ignite
Observability
Cloudstack
Geospatial
Graph
Camel/Integration
Flagon
Tomcat
Cassandra
Groovy
Web/httpd
General/Other

The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).

Submit early, submit often, at https://apachecon.com/acna2020/cfp

Rich, for the ApacheCon Planners

[jira] [Created] (KAFKA-9461) Limit DEBUG statement size when logging failed record value

[Nicolas Guyomar (Jira)]

Nicolas Guyomar created KAFKA-9461:
--

 Summary: Limit DEBUG statement size when logging failed record 
value
 Key: KAFKA-9461
 URL: https://issues.apache.org/jira/browse/KAFKA-9461
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 2.4.0
Reporter: Nicolas Guyomar


Hi,

It is possible with the current implementation that we log a full record 
content at DEBUG level, which can overwhelmed log4j buffer and OOM it : 

That stack trace was due to a 70MB messages refused by a broker

 
{code:java}
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at 
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuffer.append(StringBuffer.java:270)
at 
org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.format(PatternParser.java:419)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at 
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252)
at 
org.apache.kafka.connect.runtime.WorkerSourceTask$1.onCompletion(WorkerSourceTask.java:330){code}
 

 

Would it make sense to protect Connect directly in the ConnectRecord toString() 
method and set a configurable limit ? 

 

Thank you

 

 

 
[https://github.com/apache/kafka/blob/da4337271ef0b72643c0cf47ae51e69f79cf1901/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L348]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Ron Dagostino]

Sure, go for it.

> On Jan 21, 2020, at 8:05 AM, Николай Ижиков  wrote:
> 
> Hello, Ron.
> 
> Let’s start vote right now.
> What do you think?
> 
>> 21 янв. 2020 г., в 15:48, Ron Dagostino  написал(а):
>> 
>> LGTM.  The KIP freeze for 2.5 is officially upon us tomorrow, but hopefully 
>> this is such a simple and straightforward change with obvious security 
>> benefits that it can be added anyway.  I would put it up for a vote very 
>> quickly — tomorrow at the latest.
>> 
>> Ron
>> 
>>> On Jan 21, 2020, at 7:38 AM, Николай Ижиков  wrote:
>>> 
>>> Hello.
>>> 
>>> KIP [1] updated.
>>> Only TLSv1.2 will be enabled by default, as Rajini suggested.
>>> 
>>> Any objections to it?
>>> 
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
>>> 
>>> 
 17 янв. 2020 г., в 14:56, Николай Ижиков  
 написал(а):
 
 Thanks, Rajini.
 
 Will do it, shortly.
 
> 17 янв. 2020 г., в 14:50, Rajini Sivaram  
> написал(а):
> 
> Hi Nikolay,
> 
> 1) You can update KIP-553 to disable old protocols. This would mean:
> 1a) SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS would be just TLSv1.2
> 1b) SslConfigs.DEFAULT_SSL_PROTOCOL would become TLSv1.2
> 
> 2) When the testing for TLSv1.3 has been done, open a new KIP to enable
> TLSv1.3 by default. This would mean adding TLSv1.3 to
> SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS.
> 
> 
>> On Fri, Jan 17, 2020 at 11:40 AM Николай Ижиков  
>> wrote:
>> 
>> Hello, Rajini.
>> 
>> Yes, we can!
>> 
>> I have to write another KIP that goal will be keep only TLSv1.2 and
>> TLSv1.3 in SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS
>> Is it correct?
>> 
>> 
>>> 17 янв. 2020 г., в 14:13, Rajini Sivaram 
>> написал(а):
>>> 
>>> Hi Nikolay,
>>> 
>>> Can we split this KIP into two:
>>> 1) Remove insecure TLS protocols from the default values
>>> 2) Enable TLSv1.3
>>> 
>>> Since we are coming up to KIP freeze for 2.5.0 release, it will be good
>> if
>>> we can get at least the first one into 2.5.0. It would be a much smaller
>>> change and won't get blocked behind TLSv1.3 testing.
>>> 
>>> Thank you,
>>> 
>>> Rajini
>>> 
>>> On Tue, Jan 7, 2020 at 11:49 AM Rajini Sivaram 
>>> wrote:
>>> 
 Hi Nikolay,
 
 There a couple of things you could do:
 
 1) Run all system tests that use SSL with TLSv1.3. I had run a subset,
>> but
 it will be good to run all of them. You can do this locally using 
 docker
 with JDK 11 by updating the files in tests/docker. You will need to
>> update
 tests/kafkatest/services/security/security_config.py to enable only
 TLSv1.3. Instructions for running system tests using docker are in
 https://github.com/apache/kafka/blob/trunk/tests/README.md.
 2) For integration tests, we run a small number of tests using TLSv1.3
>> if
 the tests are run using JDK 11 and above. We need to do this for system
 tests as well. There is an open JIRA:
 https://issues.apache.org/jira/browse/KAFKA-9319. Feel free to assign
 this to yourself if you have time to do this.
 
 Regards,
 
 Rajini
 
 
 On Tue, Jan 7, 2020 at 5:15 AM Николай Ижиков 
>> wrote:
 
> Hello, Rajini.
> 
> Can you, please, clarify, what should be done?
> I can try to do tests by myself.
> 
>> 6 янв. 2020 г., в 21:29, Rajini Sivaram 
> написал(а):
>> 
>> Hi Brajesh.
>> 
>> No one is working on this yet, but will follow up with the Confluent
> tools
>> team to see when this can be done.
>> 
>> On Mon, Jan 6, 2020 at 3:29 PM Brajesh Kumar 
> wrote:
>> 
>>> Hello Rajini,
>>> 
>>> What is the plan to run system tests using JDK 11? Is someone 
>>> working
> on
>>> this?
>>> 
>>> On Mon, Jan 6, 2020 at 3:00 PM Rajini Sivaram <
>> rajinisiva...@gmail.com
>> 
>>> wrote:
>>> 
 Hi Nikolay,
 
 We can leave the KIP open and restart the discussion once system
>> tests
>>> are
 running.
 
 Thanks,
 
 Rajini
 
 On Mon, Jan 6, 2020 at 2:46 PM Николай Ижиков 
>>> wrote:
 
> Hello, Rajini.
> 
> Thanks, for the feedback.
> 
> Should I mark this KIP as declined?
> Or just wait for the system tests results?
> 
>> 6 янв. 2020 г., в 17:26, Rajini Sivaram 
> написал(а):
>> 
>> Hi Nikolay,
>> 
>> Thanks for 

Re: [DISCUSS] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Николай Ижиков]

Hello, Ron.

Let’s start vote right now.
What do you think?

> 21 янв. 2020 г., в 15:48, Ron Dagostino  написал(а):
> 
> LGTM.  The KIP freeze for 2.5 is officially upon us tomorrow, but hopefully 
> this is such a simple and straightforward change with obvious security 
> benefits that it can be added anyway.  I would put it up for a vote very 
> quickly — tomorrow at the latest.
> 
> Ron
> 
>> On Jan 21, 2020, at 7:38 AM, Николай Ижиков  wrote:
>> 
>> Hello.
>> 
>> KIP [1] updated.
>> Only TLSv1.2 will be enabled by default, as Rajini suggested.
>> 
>> Any objections to it?
>> 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
>> 
>> 
>>> 17 янв. 2020 г., в 14:56, Николай Ижиков  
>>> написал(а):
>>> 
>>> Thanks, Rajini.
>>> 
>>> Will do it, shortly.
>>> 
 17 янв. 2020 г., в 14:50, Rajini Sivaram  
 написал(а):
 
 Hi Nikolay,
 
 1) You can update KIP-553 to disable old protocols. This would mean:
 1a) SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS would be just TLSv1.2
 1b) SslConfigs.DEFAULT_SSL_PROTOCOL would become TLSv1.2
 
 2) When the testing for TLSv1.3 has been done, open a new KIP to enable
 TLSv1.3 by default. This would mean adding TLSv1.3 to
 SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS.
 
 
> On Fri, Jan 17, 2020 at 11:40 AM Николай Ижиков  
> wrote:
> 
> Hello, Rajini.
> 
> Yes, we can!
> 
> I have to write another KIP that goal will be keep only TLSv1.2 and
> TLSv1.3 in SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS
> Is it correct?
> 
> 
>> 17 янв. 2020 г., в 14:13, Rajini Sivaram 
> написал(а):
>> 
>> Hi Nikolay,
>> 
>> Can we split this KIP into two:
>> 1) Remove insecure TLS protocols from the default values
>> 2) Enable TLSv1.3
>> 
>> Since we are coming up to KIP freeze for 2.5.0 release, it will be good
> if
>> we can get at least the first one into 2.5.0. It would be a much smaller
>> change and won't get blocked behind TLSv1.3 testing.
>> 
>> Thank you,
>> 
>> Rajini
>> 
>> On Tue, Jan 7, 2020 at 11:49 AM Rajini Sivaram 
>> wrote:
>> 
>>> Hi Nikolay,
>>> 
>>> There a couple of things you could do:
>>> 
>>> 1) Run all system tests that use SSL with TLSv1.3. I had run a subset,
> but
>>> it will be good to run all of them. You can do this locally using docker
>>> with JDK 11 by updating the files in tests/docker. You will need to
> update
>>> tests/kafkatest/services/security/security_config.py to enable only
>>> TLSv1.3. Instructions for running system tests using docker are in
>>> https://github.com/apache/kafka/blob/trunk/tests/README.md.
>>> 2) For integration tests, we run a small number of tests using TLSv1.3
> if
>>> the tests are run using JDK 11 and above. We need to do this for system
>>> tests as well. There is an open JIRA:
>>> https://issues.apache.org/jira/browse/KAFKA-9319. Feel free to assign
>>> this to yourself if you have time to do this.
>>> 
>>> Regards,
>>> 
>>> Rajini
>>> 
>>> 
>>> On Tue, Jan 7, 2020 at 5:15 AM Николай Ижиков 
> wrote:
>>> 
 Hello, Rajini.
 
 Can you, please, clarify, what should be done?
 I can try to do tests by myself.
 
> 6 янв. 2020 г., в 21:29, Rajini Sivaram 
 написал(а):
> 
> Hi Brajesh.
> 
> No one is working on this yet, but will follow up with the Confluent
 tools
> team to see when this can be done.
> 
> On Mon, Jan 6, 2020 at 3:29 PM Brajesh Kumar 
 wrote:
> 
>> Hello Rajini,
>> 
>> What is the plan to run system tests using JDK 11? Is someone working
 on
>> this?
>> 
>> On Mon, Jan 6, 2020 at 3:00 PM Rajini Sivaram <
> rajinisiva...@gmail.com
> 
>> wrote:
>> 
>>> Hi Nikolay,
>>> 
>>> We can leave the KIP open and restart the discussion once system
> tests
>> are
>>> running.
>>> 
>>> Thanks,
>>> 
>>> Rajini
>>> 
>>> On Mon, Jan 6, 2020 at 2:46 PM Николай Ижиков 
>> wrote:
>>> 
 Hello, Rajini.
 
 Thanks, for the feedback.
 
 Should I mark this KIP as declined?
 Or just wait for the system tests results?
 
> 6 янв. 2020 г., в 17:26, Rajini Sivaram 
 написал(а):
> 
> Hi Nikolay,
> 
> Thanks for the KIP. We currently run system tests using JDK 8 and
>> hence
 we
> don't yet have full system test results with TLS 1.3 which
> requires
>> JDK
 11.
> We should wait until that is done before 

Re: [DISCUSS] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Ron Dagostino]

LGTM.  The KIP freeze for 2.5 is officially upon us tomorrow, but hopefully 
this is such a simple and straightforward change with obvious security benefits 
that it can be added anyway.  I would put it up for a vote very quickly — 
tomorrow at the latest.

Ron

> On Jan 21, 2020, at 7:38 AM, Николай Ижиков  wrote:
> 
> Hello.
> 
> KIP [1] updated.
> Only TLSv1.2 will be enabled by default, as Rajini suggested.
> 
> Any objections to it?
> 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> 
> 
>> 17 янв. 2020 г., в 14:56, Николай Ижиков  написал(а):
>> 
>> Thanks, Rajini.
>> 
>> Will do it, shortly.
>> 
>>> 17 янв. 2020 г., в 14:50, Rajini Sivaram  
>>> написал(а):
>>> 
>>> Hi Nikolay,
>>> 
>>> 1) You can update KIP-553 to disable old protocols. This would mean:
>>> 1a) SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS would be just TLSv1.2
>>> 1b) SslConfigs.DEFAULT_SSL_PROTOCOL would become TLSv1.2
>>> 
>>> 2) When the testing for TLSv1.3 has been done, open a new KIP to enable
>>> TLSv1.3 by default. This would mean adding TLSv1.3 to
>>> SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS.
>>> 
>>> 
 On Fri, Jan 17, 2020 at 11:40 AM Николай Ижиков  
 wrote:
 
 Hello, Rajini.
 
 Yes, we can!
 
 I have to write another KIP that goal will be keep only TLSv1.2 and
 TLSv1.3 in SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS
 Is it correct?
 
 
> 17 янв. 2020 г., в 14:13, Rajini Sivaram 
 написал(а):
> 
> Hi Nikolay,
> 
> Can we split this KIP into two:
> 1) Remove insecure TLS protocols from the default values
> 2) Enable TLSv1.3
> 
> Since we are coming up to KIP freeze for 2.5.0 release, it will be good
 if
> we can get at least the first one into 2.5.0. It would be a much smaller
> change and won't get blocked behind TLSv1.3 testing.
> 
> Thank you,
> 
> Rajini
> 
> On Tue, Jan 7, 2020 at 11:49 AM Rajini Sivaram 
> wrote:
> 
>> Hi Nikolay,
>> 
>> There a couple of things you could do:
>> 
>> 1) Run all system tests that use SSL with TLSv1.3. I had run a subset,
 but
>> it will be good to run all of them. You can do this locally using docker
>> with JDK 11 by updating the files in tests/docker. You will need to
 update
>> tests/kafkatest/services/security/security_config.py to enable only
>> TLSv1.3. Instructions for running system tests using docker are in
>> https://github.com/apache/kafka/blob/trunk/tests/README.md.
>> 2) For integration tests, we run a small number of tests using TLSv1.3
 if
>> the tests are run using JDK 11 and above. We need to do this for system
>> tests as well. There is an open JIRA:
>> https://issues.apache.org/jira/browse/KAFKA-9319. Feel free to assign
>> this to yourself if you have time to do this.
>> 
>> Regards,
>> 
>> Rajini
>> 
>> 
>> On Tue, Jan 7, 2020 at 5:15 AM Николай Ижиков 
 wrote:
>> 
>>> Hello, Rajini.
>>> 
>>> Can you, please, clarify, what should be done?
>>> I can try to do tests by myself.
>>> 
 6 янв. 2020 г., в 21:29, Rajini Sivaram 
>>> написал(а):
 
 Hi Brajesh.
 
 No one is working on this yet, but will follow up with the Confluent
>>> tools
 team to see when this can be done.
 
 On Mon, Jan 6, 2020 at 3:29 PM Brajesh Kumar 
>>> wrote:
 
> Hello Rajini,
> 
> What is the plan to run system tests using JDK 11? Is someone working
>>> on
> this?
> 
> On Mon, Jan 6, 2020 at 3:00 PM Rajini Sivaram <
 rajinisiva...@gmail.com
 
> wrote:
> 
>> Hi Nikolay,
>> 
>> We can leave the KIP open and restart the discussion once system
 tests
> are
>> running.
>> 
>> Thanks,
>> 
>> Rajini
>> 
>> On Mon, Jan 6, 2020 at 2:46 PM Николай Ижиков 
> wrote:
>> 
>>> Hello, Rajini.
>>> 
>>> Thanks, for the feedback.
>>> 
>>> Should I mark this KIP as declined?
>>> Or just wait for the system tests results?
>>> 
 6 янв. 2020 г., в 17:26, Rajini Sivaram 
>>> написал(а):
 
 Hi Nikolay,
 
 Thanks for the KIP. We currently run system tests using JDK 8 and
> hence
>>> we
 don't yet have full system test results with TLS 1.3 which
 requires
> JDK
>>> 11.
 We should wait until that is done before enabling TLS1.3 by
 default.
 
 Regards,
 
 Rajini
 
 
 On Mon, Dec 30, 2019 at 5:36 AM Николай Ижиков <
 nizhi...@apache.org
 
>>> wrote:
 
> Hello, Team.

[jira] [Created] (KAFKA-9460) Enable TLSv1.2 by default and disable all others protocol versions

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created KAFKA-9460:
--

 Summary: Enable TLSv1.2 by default and disable all others protocol 
versions
 Key: KAFKA-9460
 URL: https://issues.apache.org/jira/browse/KAFKA-9460
 Project: Kafka
  Issue Type: Improvement
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov


In KAFKA-7251 support of TLS1.3 was introduced.

For now, only TLS1.2 and TLS1.3 are recommended for the usage, other versions 
of TLS considered as obsolete:

https://www.rfc-editor.org/info/rfc8446
https://en.wikipedia.org/wiki/Transport_Layer_Security#History_and_development
But testing of TLS1.3 incomplete, for now.

We should enable actual versions of the TLS protocol by default to provide to 
the users only secure implementations.

Users can enable obsolete versions of the TLS with the configuration if they 
want to. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[Николай Ижиков]

Hello.

KIP [1] updated.
Only TLSv1.2 will be enabled by default, as Rajini suggested.

Any objections to it?

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956


> 17 янв. 2020 г., в 14:56, Николай Ижиков  написал(а):
> 
> Thanks, Rajini.
> 
> Will do it, shortly.
> 
>> 17 янв. 2020 г., в 14:50, Rajini Sivaram  
>> написал(а):
>> 
>> Hi Nikolay,
>> 
>> 1) You can update KIP-553 to disable old protocols. This would mean:
>>  1a) SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS would be just TLSv1.2
>>  1b) SslConfigs.DEFAULT_SSL_PROTOCOL would become TLSv1.2
>> 
>> 2) When the testing for TLSv1.3 has been done, open a new KIP to enable
>> TLSv1.3 by default. This would mean adding TLSv1.3 to
>> SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS.
>> 
>> 
>> On Fri, Jan 17, 2020 at 11:40 AM Николай Ижиков  wrote:
>> 
>>> Hello, Rajini.
>>> 
>>> Yes, we can!
>>> 
>>> I have to write another KIP that goal will be keep only TLSv1.2 and
>>> TLSv1.3 in SslConfigs.DEFAULT_SSL_ENABLED_PROTOCOLS
>>> Is it correct?
>>> 
>>> 
 17 янв. 2020 г., в 14:13, Rajini Sivaram 
>>> написал(а):
 
 Hi Nikolay,
 
 Can we split this KIP into two:
 1) Remove insecure TLS protocols from the default values
 2) Enable TLSv1.3
 
 Since we are coming up to KIP freeze for 2.5.0 release, it will be good
>>> if
 we can get at least the first one into 2.5.0. It would be a much smaller
 change and won't get blocked behind TLSv1.3 testing.
 
 Thank you,
 
 Rajini
 
 On Tue, Jan 7, 2020 at 11:49 AM Rajini Sivaram 
 wrote:
 
> Hi Nikolay,
> 
> There a couple of things you could do:
> 
> 1) Run all system tests that use SSL with TLSv1.3. I had run a subset,
>>> but
> it will be good to run all of them. You can do this locally using docker
> with JDK 11 by updating the files in tests/docker. You will need to
>>> update
> tests/kafkatest/services/security/security_config.py to enable only
> TLSv1.3. Instructions for running system tests using docker are in
> https://github.com/apache/kafka/blob/trunk/tests/README.md.
> 2) For integration tests, we run a small number of tests using TLSv1.3
>>> if
> the tests are run using JDK 11 and above. We need to do this for system
> tests as well. There is an open JIRA:
> https://issues.apache.org/jira/browse/KAFKA-9319. Feel free to assign
> this to yourself if you have time to do this.
> 
> Regards,
> 
> Rajini
> 
> 
> On Tue, Jan 7, 2020 at 5:15 AM Николай Ижиков 
>>> wrote:
> 
>> Hello, Rajini.
>> 
>> Can you, please, clarify, what should be done?
>> I can try to do tests by myself.
>> 
>>> 6 янв. 2020 г., в 21:29, Rajini Sivaram 
>> написал(а):
>>> 
>>> Hi Brajesh.
>>> 
>>> No one is working on this yet, but will follow up with the Confluent
>> tools
>>> team to see when this can be done.
>>> 
>>> On Mon, Jan 6, 2020 at 3:29 PM Brajesh Kumar 
>> wrote:
>>> 
 Hello Rajini,
 
 What is the plan to run system tests using JDK 11? Is someone working
>> on
 this?
 
 On Mon, Jan 6, 2020 at 3:00 PM Rajini Sivaram <
>>> rajinisiva...@gmail.com
>>> 
 wrote:
 
> Hi Nikolay,
> 
> We can leave the KIP open and restart the discussion once system
>>> tests
 are
> running.
> 
> Thanks,
> 
> Rajini
> 
> On Mon, Jan 6, 2020 at 2:46 PM Николай Ижиков 
 wrote:
> 
>> Hello, Rajini.
>> 
>> Thanks, for the feedback.
>> 
>> Should I mark this KIP as declined?
>> Or just wait for the system tests results?
>> 
>>> 6 янв. 2020 г., в 17:26, Rajini Sivaram 
>> написал(а):
>>> 
>>> Hi Nikolay,
>>> 
>>> Thanks for the KIP. We currently run system tests using JDK 8 and
 hence
>> we
>>> don't yet have full system test results with TLS 1.3 which
>>> requires
 JDK
>> 11.
>>> We should wait until that is done before enabling TLS1.3 by
>>> default.
>>> 
>>> Regards,
>>> 
>>> Rajini
>>> 
>>> 
>>> On Mon, Dec 30, 2019 at 5:36 AM Николай Ижиков <
>>> nizhi...@apache.org
>>> 
>> wrote:
>>> 
 Hello, Team.
 
 Any feedback on this KIP?
 Do we need this in Kafka?
 
> 24 дек. 2019 г., в 18:28, Nikolay Izhikov 
 написал(а):
> 
> Hello,
> 
> I'd like to start a discussion of KIP.
> Its goal is to enable TLSv1.3 and disable obsolete versions by
> default.
> 
> 
 
>> 
> 
 
>> 
>>> 

[jira] [Created] (KAFKA-9459) MM2 sync topic config does work

[Badai Aqrandista (Jira)]

Badai Aqrandista created KAFKA-9459:
---

 Summary: MM2 sync topic config does work
 Key: KAFKA-9459
 URL: https://issues.apache.org/jira/browse/KAFKA-9459
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.4.0
Reporter: Badai Aqrandista


I have MM2 configured as follow:

{code:java}
{
"name": "mm2-from-1-to-2",
"config": {
  
"connector.class":"org.apache.kafka.connect.mirror.MirrorSourceConnector",
  "topics":"foo",
  "key.converter": 
"org.apache.kafka.connect.converters.ByteArrayConverter",
  "value.converter": 
"org.apache.kafka.connect.converters.ByteArrayConverter",
  "sync.topic.configs.enabled":"true",
  "sync.topic.configs.interval.seconds": 60,
  "sync.topic.acls.enabled": "false",
  "replication.factor": 1,
  "offset-syncs.topic.replication.factor": 1,
  "heartbeats.topic.replication.factor": 1,
  "checkpoints.topic.replication.factor": 1,

  "target.cluster.alias":"dest",
  "target.cluster.bootstrap.servers":"dest.example.com:9092",

  "source.cluster.alias":"src",
  "source.cluster.bootstrap.servers":"src.example.com:9092",

  "tasks.max": 1}
}
{code}

Topic "foo" is configured with "cleanup.policy=compact". But after waiting for 
15 minutes, I still don't see "src.foo" in the destination cluster has 
"cleanup.policy=compact".

I had the connect node to run in TRACE level and I could not find any calls to 
describeConfigs 
(https://github.com/apache/kafka/blob/2.4.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L327).
 This implies it never actually get a list of topics that it needs to get topic 
configs from.

And I am suspecting this code 
(https://github.com/apache/kafka/blob/2.4.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L214-L220):


{code:java}
private Set topicsBeingReplicated() {
return knownTopicPartitions.stream()
.map(x -> x.topic())
.distinct()
.filter(x -> knownTargetTopics.contains(formatRemoteTopic(x)))
.collect(Collectors.toSet());
}
{code}

knownTopicPartitions contains topic-partitions from the source cluster.
knownTargetTopics contains topic-partitions from the target cluster, whose 
topic names contain source alias already.

So, why is topicsBeingReplicated (list of topic-partitions from source cluster) 
being filtered using knownTopicPartitions (list of topic-partitions from target 
cluster)?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9458) Kafka crashed in windows environment

[hirik (Jira)]

hirik created KAFKA-9458:


 Summary: Kafka crashed in windows environment
 Key: KAFKA-9458
 URL: https://issues.apache.org/jira/browse/KAFKA-9458
 Project: Kafka
  Issue Type: Bug
 Environment: Windows 10
Reporter: hirik
 Attachments: logs.zip

Hi,

while I was trying to validate Kafka retention policy, Kafka Server crashed 
with below exception trace. 

[2020-01-21 17:10:40,475] INFO [Log partition=test1-3, 
dir=C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka] Rolled 
new log segment at offset 1 in 52 ms. (kafka.log.Log)
[2020-01-21 17:10:40,484] ERROR Error while deleting segments for test1-3 in 
dir C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka 
(kafka.server.LogDirFailureChannel)
java.nio.file.FileSystemException: 
C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex
 -> 
C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex.deleted:
 The process cannot access the file because it is being used by another process.

at 
java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
 at 
java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
 at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:395)
 at 
java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:292)
 at java.base/java.nio.file.Files.move(Files.java:1425)
 at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:795)
 at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:209)
 at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:497)
 at kafka.log.Log.$anonfun$deleteSegmentFiles$1(Log.scala:2206)
 at kafka.log.Log.$anonfun$deleteSegmentFiles$1$adapted(Log.scala:2206)
 at scala.collection.immutable.List.foreach(List.scala:305)
 at kafka.log.Log.deleteSegmentFiles(Log.scala:2206)
 at kafka.log.Log.removeAndDeleteSegments(Log.scala:2191)
 at kafka.log.Log.$anonfun$deleteSegments$2(Log.scala:1700)
 at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.scala:17)
 at kafka.log.Log.maybeHandleIOException(Log.scala:2316)
 at kafka.log.Log.deleteSegments(Log.scala:1691)
 at kafka.log.Log.deleteOldSegments(Log.scala:1686)
 at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1763)
 at kafka.log.Log.deleteOldSegments(Log.scala:1753)
 at kafka.log.LogManager.$anonfun$cleanupLogs$3(LogManager.scala:982)
 at kafka.log.LogManager.$anonfun$cleanupLogs$3$adapted(LogManager.scala:979)
 at scala.collection.immutable.List.foreach(List.scala:305)
 at kafka.log.LogManager.cleanupLogs(LogManager.scala:979)
 at kafka.log.LogManager.$anonfun$startup$2(LogManager.scala:403)
 at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:116)
 at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:65)
 at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
 at 
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 at java.base/java.lang.Thread.run(Thread.java:830)
 Suppressed: java.nio.file.FileSystemException: 
C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex
 -> 
C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka\test1-3\.timeindex.deleted:
 The process cannot access the file because it is being used by another process.

at 
java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
 at 
java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
 at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:309)
 at 
java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:292)
 at java.base/java.nio.file.Files.move(Files.java:1425)
 at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:792)
 ... 27 more
[2020-01-21 17:10:40,495] INFO [ReplicaManager broker=0] Stopping serving 
replicas in dir 
C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka 
(kafka.server.ReplicaManager)
[2020-01-21 17:10:40,495] ERROR Uncaught exception in scheduled task 
'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting 
segments for test1-3 in dir 
C:\Users\Administrator\Downloads\kafka\bin\windows\..\..\data\kafka
Caused by: java.nio.file.FileSystemException: 

[jira] [Created] (KAFKA-9457) Flaky test org.apache.kafka.common.network.SelectorTest.testGracefulClose

[Rajini Sivaram (Jira)]

Rajini Sivaram created KAFKA-9457:
-

 Summary: Flaky test 
org.apache.kafka.common.network.SelectorTest.testGracefulClose
 Key: KAFKA-9457
 URL: https://issues.apache.org/jira/browse/KAFKA-9457
 Project: Kafka
  Issue Type: Bug
  Components: network
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.5.0


org.apache.kafka.common.network.SelectorTest.testGracefulClose has been failing 
a lot in PR builds:

{{java.lang.AssertionError: expected:<1> but was:<0>}}
{{ at org.junit.Assert.fail(Assert.java:89)}}
{{ at org.junit.Assert.failNotEquals(Assert.java:835)}}
{{ at org.junit.Assert.assertEquals(Assert.java:647)}}
{{ at org.junit.Assert.assertEquals(Assert.java:633)}}
{{ at 
org.apache.kafka.common.network.SelectorTest.testGracefulClose(SelectorTest.java:588)}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-565: Using AclCommand,avoid call the global method loadcache in SimpleAclAuthorizer

[Rajini Sivaram]

Hi Steven,

Thanks for the KIP. A few questions/comments:

1) The command line option for AclCommand makes it the user's
responsibility to determine whether cache should be loaded. That doesn't
feel like a good idea. If you are listing ACLs, you need the cache. More
importantly, you need the cache for some code paths in delete and that
could be authorizer-dependent. It feels dangerous to make that a choice
when the result of not doing so would potentially retain ACLs that you
didn't intend to.

2) Even though the KIP talks about the deprecated SimpleAclAuthorizer, I
guess you also mean the new AclAuthorizer since the PR updates the new one.
We should clarify in the KIP.

3) The recommended way to update ACLs is using --bootstrap-server option
for AclCommand which uses the Kafka protocol to talk to brokers and the
update is performed by brokers which already have all ACLs loaded into
their cache. In case you have found issues with this approach, it will be
good to understand what the issues are so that we can improve this path.

On Tue, Jan 21, 2020 at 1:50 AM Steven Lu  wrote:

> Hello all,
>
> In the class Named AclCommand,configure SimpleAclAuthorizer,but no need
> call loadCache.
> now we have 20,000 topics in kafka cluster,everytime I run AclCommand,all
> these topics's Alcs need to be authed, it will be very slow.
> The purpose of this optimization is：we can choose to not load the acl of
> all topics into memory, mainly for adding and deleting permissions.
>
> PR Available here: https://github.com/apache/kafka/pull/7706
> KIP Available here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-565%3A+Using+AclCommand%2Cavoid+call+the+global+method+loadcache+in+SimpleAclAuthorizer
> Issue Available here: https://issues.apache.org/jira/browse/KAFKA-9424
>
> mainly for adding and deleting permissions,we can choose to not load the
> acl of all topics into memory,then we can add two args "--load-acl-cache"
> "false" in AclCommand.main;else you don't add these args, it will load the
> acl cache defaultly.
>
> we can choose improve the running time from minutes to less than one
> second.
>
> Thanks,
> Steven
>