Build failed in Jenkins: kafka-2.1-jdk8 #224

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8391; Improved the Connect integration tests to make them less

--
[...truncated 926.82 KB...]

kafka.utils.CoreUtilsTest > testReadInt STARTED

kafka.utils.CoreUtilsTest > testReadInt PASSED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate STARTED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate PASSED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.CoreUtilsTest > testCsvMap STARTED

kafka.utils.CoreUtilsTest > testCsvMap PASSED

kafka.utils.CoreUtilsTest > testInLock STARTED

kafka.utils.CoreUtilsTest > testInLock PASSED

kafka.utils.CoreUtilsTest > testTryAll STARTED

kafka.utils.CoreUtilsTest > testTryAll PASSED

kafka.utils.CoreUtilsTest > testSwallow STARTED

kafka.utils.CoreUtilsTest > testSwallow PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.JsonTest > testParseToWithInvalidJson STARTED

kafka.utils.JsonTest > testParseToWithInvalidJson PASSED

kafka.utils.JsonTest > testParseTo STARTED

kafka.utils.JsonTest > testParseTo PASSED

kafka.utils.JsonTest > testJsonParse STARTED

kafka.utils.JsonTest > testJsonParse PASSED

kafka.utils.JsonTest > testLegacyEncodeAsString STARTED

kafka.utils.JsonTest > testLegacyEncodeAsString PASSED

kafka.utils.JsonTest > testEncodeAsBytes STARTED

kafka.utils.JsonTest > testEncodeAsBytes PASSED

kafka.utils.JsonTest > testEncodeAsString STARTED

kafka.utils.JsonTest > testEncodeAsString PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters STARTED

kafka.utils.CommandLineUtilsTest > testParseArgsWithMultipleDelimiters PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgWithNoDelimiter PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

Re: UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

[James Cheng]

Alexandre,

You are right that this is a problem. There is a JIRA on this from a while 
back. 

https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4680

I don’t think anyone is currently working on it right now. 

-James

Sent from my iPhone

> On Aug 13, 2019, at 1:17 AM, Alexandre Dupriez  
> wrote:
> 
> Hello all,
> 
> We run into a scenario where we had misconfigured the replication factor
> and the minimum in-sync replicas count in such a way that the replication
> factor (either default or defined at the topic level) is strictly lower
> than the property min.insync.replicas.
> 
> We observed broker metrics reporting UnderReplicatedPartitions = 0 and
> UnderMinPartitionIsrCount > 0, and the topic’s partitions were unavailable
> for producers (with ack=all) and consumers.
> 
> Since it seems to be impossible in this scenario to ever reach the number
> of in-sync replicas, making partitions permanently unavailable, it could be
> worth to prevent this misconfiguration to make its way to the broker, e.g.
> a check could be added when a topic is created to ensure the replication
> factor is greater than or equals to the minimum number of in-sync replicas.
> 
> I may have missed something though. What do you think?
> 
> Thank you,
> Alexandre

Jenkins build is back to normal : kafka-2.0-jdk8 #289

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

[Satish Duggana]

Hi Colin,
Thanks for the KIP. Optional fields and var length encoding support is a great
improvement for the protocol.

>>Optional fields can have any type, except that they cannot be arrays.
Note that the restriction against having tagged arrays is just to simplify
serialization.  We can relax this restriction in the future without changing
the protocol on the wire.

Can an Optional field have a struct type which internally contains an array
field at any level?

Thanks,
Satish.



On Tue, Aug 13, 2019 at 11:49 PM David Jacot  wrote:
>
> Hi Colin,
>
> Thank you for the KIP! Things are well explained!. It is huge improvement
> for the Kafka protocol. I have few comments on the proposal:
>
> 1. The interleaved tag/length header sounds like a great optimisation as it
> would be shorter on average. The downside, as
> you already pointed out, is that it makes the decoding and the specs more
> complex. Personally, I would also favour using two
> vaints in this particular case to keep things simple.
>
> 2. As discussed, I wonder if it would make sense to extend to KIP to also
> support optional fields in the Record Header. I think
> that it could be interesting to have such capability for common fields
> across all the requests or responses (e.g. tracing id).
>
> Regards,
> David
>
>
>
> On Tue, Aug 13, 2019 at 10:00 AM Jason Gustafson  wrote:
>
> > > Right, I was planning on doing exactly that for all the auto-generated
> > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > better use of time to convert the manual ones to auto gen first (with the
> > possible exception of Fetch/Produce, where the ROI may be higher for the
> > manual work)
> >
> > Yeah, that makes sense. Maybe we can include the version bump for all RPCs
> > in this KIP, but we can implement it lazily as the protocols are converted.
> >
> > -Jason
> >
> > On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe  wrote:
> >
> > > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote:
> > > > Hi Colin,
> > > >
> > > > Thanks for the KIP! This is a significant improvement. One of my
> > personal
> > > > interests in this proposal is solving the compatibility problems we
> > have
> > > > with the internal schemas used to define consumer offsets and
> > transaction
> > > > metadata. Currently we have to guard schema bumps with the inter-broker
> > > > protocol format. Once the format is bumped, there is no way to
> > downgrade.
> > > > By fixing this, we can potentially begin using the new schemas before
> > the
> > > > IBP is bumped while still allowing downgrade.
> > > >
> > > > There are a surprising number of other situations we have encountered
> > > this
> > > > sort of problem. We have hacked around it in special cases by allowing
> > > > nullable fields to the end of the schema, but this is not really an
> > > > extensible approach. I'm looking forward to having a better option.
> > >
> > > Yeah, this problem keeps coming up.
> > >
> > > >
> > > > With that said, I have a couple questions on the proposal:
> > > >
> > > > 1. For each request API, we need one version bump to begin support for
> > > > "flexible versions." Until then, we won't have the option of using
> > tagged
> > > > fields even if the broker knows how to handle them. Does it make sense
> > to
> > > > go ahead and do a universal bump of each request API now so that we'll
> > > have
> > > > this option going forward?
> > >
> > > Right, I was planning on doing exactly that for all the auto-generated
> > > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > > better use of time to convert the manual ones to auto gen first (with the
> > > possible exception of Fetch/Produce, where the ROI may be higher for the
> > > manual work)
> > >
> > > > 2. The alternating length/tag header encoding lets us save a byte in
> > the
> > > > common case. The downside is that it's a bit more complex to specify.
> > It
> > > > also has some extra cost if the length exceeds the tag substantially.
> > > > Basically we'd have to pad the tag, right? I think I'm wondering if we
> > > > should just bite the bullet and use two varints instead.
> > >
> > > That’s a fair point. It would be shorter on average, but worse for some
> > > exceptional cases. Also, the decoding would be more complex, which might
> > be
> > > a good reason to go for just having two varints. Yeah, let’s simplify.
> > >
> > > Regards,
> > > Colin
> > >
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Fri, Aug 9, 2019 at 4:31 PM Colin McCabe 
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've made some updates to this KIP. Specifically, I wanted to avoid
> > > > > including escape bytes in the serialization format, since it was too
> > > > > complex. Also, I think this is a good opportunity to slim down our
> > > > > variable length fields.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Thu, Jul 11, 2019, at 20:52, Colin McCabe wrote:
> > > > > > 

[jira] [Created] (KAFKA-8800) Flaky Test SaslScramSslEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-8800:
--

 Summary: Flaky Test 
SaslScramSslEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
 Key: KAFKA-8800
 URL: https://issues.apache.org/jira/browse/KAFKA-8800
 Project: Kafka
  Issue Type: Bug
  Components: core, security, unit tests
Affects Versions: 2.4.0
Reporter: Matthias J. Sax


[https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6956/testReport/junit/kafka.api/SaslScramSslEndToEndAuthorizationTest/testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl/]
{quote}org.scalatest.exceptions.TestFailedException: Consumed 0 records before 
timeout instead of the expected 1 records at 
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) at 
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) at 
org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389) at 
org.scalatest.Assertions.fail(Assertions.scala:1091) at 
org.scalatest.Assertions.fail$(Assertions.scala:1087) at 
org.scalatest.Assertions$.fail(Assertions.scala:1389) at 
kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:822) at 
kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:781) at 
kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1312) at 
kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1320) at 
kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:522)
 at 
kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:361){quote}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Build failed in Jenkins: kafka-2.2-jdk8 #160

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread

[rhauch] KAFKA-8391; Improved the Connect integration tests to make them less

--
[...truncated 2.72 MB...]
kafka.api.GroupAuthorizerIntegrationTest > testUnauthorizedCreatePartitions 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testConsumeWithoutTopicDescribeAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testConsumeWithoutTopicDescribeAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnEndTransaction
 STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnEndTransaction
 PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnSendOffsetsToTxn
 STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
shouldThrowTransactionalIdAuthorizationExceptionWhenNoTransactionAccessOnSendOffsetsToTxn
 PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithNoGroupAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithNoGroupAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoDescribeTransactionalIdAcl STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testTransactionalProducerInitTransactionsNoDescribeTransactionalIdAcl PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testCreateTopicAuthorizationWithClusterCreate STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testCreateTopicAuthorizationWithClusterCreate PASSED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testAuthorizationWithTopicExisting 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testAuthorizationWithTopicExisting 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithoutDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testUnauthorizedDeleteRecordsWithoutDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testMetadataWithTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testMetadataWithTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testProduceWithTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testProduceWithTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testDescribeGroupApiWithNoGroupAcl 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testDescribeGroupApiWithNoGroupAcl 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopic STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testPatternSubscriptionMatchingInternalTopic PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testSendOffsetsWithNoConsumerGroupDescribeAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testSendOffsetsWithNoConsumerGroupDescribeAccess PASSED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchTopicDescribe STARTED

kafka.api.GroupAuthorizerIntegrationTest > testOffsetFetchTopicDescribe PASSED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicAndGroupRead 
STARTED

kafka.api.GroupAuthorizerIntegrationTest > testCommitWithTopicAndGroupRead 
PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testIdempotentProducerNoIdempotentWriteAclInInitProducerId STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testIdempotentProducerNoIdempotentWriteAclInInitProducerId PASSED

kafka.api.GroupAuthorizerIntegrationTest > 
testSimpleConsumeWithExplicitSeekAndNoGroupAccess STARTED

kafka.api.GroupAuthorizerIntegrationTest > 
testSimpleConsumeWithExplicitSeekAndNoGroupAccess PASSED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.TransactionsTest > testBasicTransactions STARTED

kafka.api.TransactionsTest > testBasicTransactions PASSED

kafka.api.TransactionsTest > testFencingOnSendOffsets STARTED

kafka.api.TransactionsTest > testFencingOnSendOffsets PASSED

kafka.api.TransactionsTest > testFencingOnAddPartitions STARTED

kafka.api.TransactionsTest > testFencingOnAddPartitions PASSED

kafka.api.TransactionsTest > 

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

[Boyang Chen]

Hey Guozhang,

thanks for the suggestion. Could you elaborate more on why defining a
direct consumer API would be easier? The benefit of reusing consumer
rebalance listener is to consolidate the entry point of consumer internal
states. Compared with letting consumer generate a deep-copy of metadata
every time we call #sendOffsetsToTransactions, using a callback seems
reducing unnecessary updates towards the metadata. WDYT?

Boyang

On Tue, Aug 13, 2019 at 2:14 PM Guozhang Wang  wrote:

> Hi Boyang, Jason,
>
> If we are going to expose the generation id / group.instance id etc anyways
> I think its slightly better to just add a new API on KafkaConsumer
> returning the ConsumerGroupMetadata (option 3) than passing it in on an
> additional callback of ConsumerRebalanceListener.
> It feels easier to leverage, than requiring users to pass in the listener.
>
> Guozhang
>
> On Mon, Aug 12, 2019 at 3:41 PM Boyang Chen 
> wrote:
>
> > Thanks Jason, the intuition behind defining a separate callback function
> is
> > that, with KIP-429 we no longer guarantee to call OnPartitionsAssigned()
> or
> > OnPartitionsRevoked() with each rebalance. Our requirement is to be
> > up-to-date with group metadata such as generation information, so
> callback
> > like onGroupJoined() would make more sense as it should be invoked after
> > every successful rebalance.
> >
> > Best,
> > Boyang
> >
> > On Mon, Aug 12, 2019 at 2:02 PM Jason Gustafson 
> > wrote:
> >
> > > Hey Boyang,
> > >
> > > I favor option 4 as well. It's a little more cumbersome than 3 for this
> > use
> > > case, but it seems like a cleaner separation of concerns. The rebalance
> > > listener is already concerned with events affecting the assignment
> > > lifecycle and group membership. I think the only thing I'm wondering is
> > > whether it should be a separate callback as you've suggested, or if it
> > > would make sense to overload `onPartitionsAssigned`. If it's separate,
> > > maybe a name like `onGroupJoined` would be clearer?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > >
> > > On Thu, Aug 8, 2019 at 10:59 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Thank you Jason. We had some offline discussion on properly keeping
> > group
> > > > metadata up to date, and here are some of our options brainstormed:
> > > > 1. Let the caller of `sendOffsetsToTransaction(offset, metadata)`
> > > maintain
> > > > the ever-changing group metadata. This could be done on stream side,
> > but
> > > > for non-stream EOS the sample code will become complicated as the
> user
> > > > needs to implement the partition assignor interface to get the update
> > > from
> > > > `onAssignment`
> > > >
> > > > 2. Get a new API on producer like `refreshGroupMetadata(metadata)`.
> > This
> > > is
> > > > similar to option 1 except that now in the partition assignor
> callback
> > we
> > > > could straightly pass in the producer instance, which simplifies the
> > > > non-stream EOS, however this new API seems weird to define on
> producer.
> > > >
> > > > 3. Make an accessing interface to group metadata, or just expose the
> > > group
> > > > metadata through a consumer API like `consumer.GroupMetadata()`. This
> > is
> > > > the old way which avoids the users’ effort to implement partition
> > > assignor
> > > > directly.
> > > >
> > > > 4. Expose the group metadata through rebalance listener, which is a
> > more
> > > > well-known and adopted callback interface. We could do sth like
> > > > `onGroupMetadataUpdated(ConsumerGroupMetadata metadata)`
> > > >
> > > > To simplify the code logic, we believe option 3 & 4 are better
> > solutions,
> > > > and of which I slightly prefer option 4 as it is the most clean
> > solution
> > > > with less intrusion to both consumer and producer APIs.
> > > >
> > > > WDYT?
> > > >
> > > > Boyang
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 7, 2019 at 9:20 AM Jason Gustafson 
> > > wrote:
> > > >
> > > > > Hi Boyang,
> > > > >
> > > > > > We already persist member.id, instance.id and generation.id in
> the
> > > > > offset
> > > > > topic, what extra fields we need to store?
> > > > >
> > > > > Yeah, you're right. I was a little confused and thought this
> > > information
> > > > > was needed by the transaction coordinator.
> > > > >
> > > > > > This should be easily done on the stream side as we have
> > > > > StreamsPartitionAssignor to reflect metadata changes upon
> > > > #onAssignment(),
> > > > > but non-stream user has to code the callback by hand, do you think
> > the
> > > > > convenience we sacrifice here worth the simplification benefit?
> > > > >
> > > > > Either way, you need a reference to the consumer. I was mostly just
> > > > > thinking it would be better to reduce the integration point to its
> > > > minimum.
> > > > > Have you thought through the implications of needing to keep
> around a
> > > > > reference to the consumer in the producer? What if it gets closed?
> It
> > > > seems
> > > > > better 

Build failed in Jenkins: kafka-trunk-jdk8 #3849

[Apache Jenkins Server]

See 


Changes:

[gwen] KAFKA-8391; Improved the Connect integration tests to make them less

[jason] MINOR: Add fetch from follower system test (#7166)

--
[...truncated 2.59 MB...]

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testCreateTask STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testCreateTask PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTasksRequest STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTasksRequest PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testCoordinatorStatus 
STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testCoordinatorStatus 
PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testCoordinatorUptime 
STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testCoordinatorUptime 
PASSED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessWithFailedExit STARTED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessWithFailedExit PASSED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessNotFound STARTED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessNotFound PASSED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessForceKillTimeout STARTED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessForceKillTimeout PASSED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > testProcessStop 
STARTED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > testProcessStop 
PASSED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessWithNormalExit STARTED

org.apache.kafka.trogdor.workload.ExternalCommandWorkerTest > 
testProcessWithNormalExit PASSED

org.apache.kafka.trogdor.workload.ThrottleTest > testThrottle STARTED

org.apache.kafka.trogdor.workload.ThrottleTest > testThrottle PASSED

org.apache.kafka.trogdor.workload.TimeIntervalTransactionsGeneratorTest > 
testCommitsTransactionAfterIntervalPasses STARTED

org.apache.kafka.trogdor.workload.TimeIntervalTransactionsGeneratorTest > 
testCommitsTransactionAfterIntervalPasses PASSED

org.apache.kafka.trogdor.workload.ConsumeBenchSpecTest > 
testMaterializeTopicsWithSomePartitions STARTED

org.apache.kafka.trogdor.workload.ConsumeBenchSpecTest > 
testMaterializeTopicsWithSomePartitions PASSED

org.apache.kafka.trogdor.workload.ConsumeBenchSpecTest > 
testMaterializeTopicsWithNoPartitions STARTED

org.apache.kafka.trogdor.workload.ConsumeBenchSpecTest > 
testMaterializeTopicsWithNoPartitions PASSED

org.apache.kafka.trogdor.workload.ConsumeBenchSpecTest > 
testInvalidTopicNameRaisesExceptionInMaterialize STARTED

org.apache.kafka.trogdor.workload.ConsumeBenchSpecTest > 
testInvalidTopicNameRaisesExceptionInMaterialize PASSED

org.apache.kafka.trogdor.workload.HistogramTest > testHistogramPercentiles 
STARTED

org.apache.kafka.trogdor.workload.HistogramTest > testHistogramPercentiles 
PASSED

org.apache.kafka.trogdor.workload.HistogramTest > testHistogramSamples STARTED

org.apache.kafka.trogdor.workload.HistogramTest > testHistogramSamples PASSED

org.apache.kafka.trogdor.workload.HistogramTest > testHistogramAverage STARTED

org.apache.kafka.trogdor.workload.HistogramTest > testHistogramAverage PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testRandomComponentPayloadGeneratorErrors STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testRandomComponentPayloadGeneratorErrors PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testConstantPayloadGenerator STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testConstantPayloadGenerator PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testSequentialPayloadGenerator STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testSequentialPayloadGenerator PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testNullPayloadGenerator STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testNullPayloadGenerator PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testUniformRandomPayloadGenerator STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testUniformRandomPayloadGenerator PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > testPayloadIterator 
STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > testPayloadIterator 
PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testUniformRandomPayloadGeneratorPaddingBytes STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testUniformRandomPayloadGeneratorPaddingBytes PASSED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 
testRandomComponentPayloadGenerator STARTED

org.apache.kafka.trogdor.workload.PayloadGeneratorTest > 

Build failed in Jenkins: kafka-2.1-jdk8 #223

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread

--
[...truncated 685.27 KB...]

kafka.api.UserQuotaTest > testProducerConsumerOverrideUnthrottled STARTED

kafka.api.UserQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.UserQuotaTest > testThrottledProducerConsumer STARTED
ERROR: Could not install GRADLE_4_8_1_HOME
java.lang.NullPointerException

kafka.api.UserQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.UserQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.UserQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.UserQuotaTest > testThrottledRequest STARTED

kafka.api.UserQuotaTest > testThrottledRequest PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclDescribe STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclDescribe PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testLegacyAclOpsNeverAffectOrReturnPrefixed STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testLegacyAclOpsNeverAffectOrReturnPrefixed PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAttemptToCreateInvalidAcls 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAttemptToCreateInvalidAcls 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclAuthorizationDenied STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclAuthorizationDenied PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations2 STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclOperations2 PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAclDelete STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAclDelete PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeReplicaLogDirs STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeReplicaLogDirs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testInvalidAlterConfigs STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testInvalidAlterConfigs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testAlterLogDirsAfterDeleteRecords STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testAlterLogDirsAfterDeleteRecords PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testConsumeAfterDeleteRecords 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testConsumeAfterDeleteRecords 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testClose STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testClose PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testReplicaCanFetchFromLogStartOffsetAfterDeleteRecords STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testReplicaCanFetchFromLogStartOffsetAfterDeleteRecords PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testMinimumRequestTimeouts STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testMinimumRequestTimeouts PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testForceClose STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testForceClose PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testListNodes STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testListNodes PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDelayedClose STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDelayedClose PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testLongTopicNames STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testLongTopicNames PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testCreateDeleteTopics STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testCreateDeleteTopics PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testOffsetsForTimesAfterDeleteRecords STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testOffsetsForTimesAfterDeleteRecords PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDeleteRecordsWithException 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDeleteRecordsWithException 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeLogDirs STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeLogDirs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAlterReplicaLogDirs STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAlterReplicaLogDirs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testLogStartOffsetCheckpoint 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testLogStartOffsetCheckpoint 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeCluster STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeCluster PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testCreatePartitions STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testCreatePartitions PASSED

[jira] [Resolved] (KAFKA-7941) Connect KafkaBasedLog work thread terminates when getting offsets fails because broker is unavailable

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-7941.
--
   Resolution: Fixed
 Reviewer: Randall Hauch
Fix Version/s: 2.3.1
   2.4.0
   2.2.2
   2.1.2
   2.0.2

Merged to the `trunk`, `2.3`, `2.2`, `2.1`, and `2.0` branches.

Thanks, [~pgwhalen]!

> Connect KafkaBasedLog work thread terminates when getting offsets fails 
> because broker is unavailable
> -
>
> Key: KAFKA-7941
> URL: https://issues.apache.org/jira/browse/KAFKA-7941
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Paul Whalen
>Assignee: Paul Whalen
>Priority: Minor
> Fix For: 2.0.2, 2.1.2, 2.2.2, 2.4.0, 2.3.1
>
>
> My team has run into this Connect bug regularly in the last six months while 
> doing infrastructure maintenance that causes intermittent broker availability 
> issues.  I'm a little surprised it exists given how routinely it affects us, 
> so perhaps someone in the know can point out if our setup is somehow just 
> incorrect.  My team is running 2.0.0 on both the broker and client, though 
> from what I can tell from reading the code, the issue continues to exist 
> through 2.2; at least, I was able to write a failing unit test that I believe 
> reproduces it.
> When a {{KafkaBasedLog}} worker thread in the Connect runtime calls 
> {{readLogToEnd}} and brokers are unavailable, the {{TimeoutException}} from 
> the consumer {{endOffsets}} call is uncaught all the way up to the top level 
> {{catch (Throwable t)}}, effectively killing the thread until restarting 
> Connect.  The result is Connect stops functioning entirely, with no 
> indication except for that log line - tasks still show as running.
> The proposed fix is to simply catch and log the {{TimeoutException}}, 
> allowing the worker thread to retry forever.
> Alternatively, perhaps there is not an expectation that Connect should be 
> able to recover following broker unavailability, though that would be 
> disappointing.  I would at least hope hope for a louder failure then the 
> single {{ERROR}} log.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Build failed in Jenkins: kafka-trunk-jdk11 #750

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197)

[rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread

--
[...truncated 6.51 MB...]

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullFieldToTime PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullFieldToUnix STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullFieldToUnix PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigNoTargetType STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigNoTargetType PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaStringToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaStringToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimeToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimeToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessUnixToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessUnixToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToString STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToString PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidFormat STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidFormat PASSED

> Task :streams:examples:test

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test 
STARTED

org.apache.kafka.streams.examples.wordcount.WordCountProcessorTest > test PASSED

> Task :streams:streams-scala:test

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > 

[jira] [Created] (KAFKA-8799) Support ability to pass global user data to consumers during Assignment

[Vinoth Chandar (JIRA)]

Vinoth Chandar created KAFKA-8799:
-

 Summary: Support ability to pass global user data to consumers 
during Assignment
 Key: KAFKA-8799
 URL: https://issues.apache.org/jira/browse/KAFKA-8799
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Vinoth Chandar


This is a follow up from KAFKA-7149 

*Background :* 

Although we reduced the size of the AssignmentInfo object sent during each 
rebalance from leader to all followers in KAFKA-7149, we still repeat the same 
_partitionsByHost_ map for each host (all this when interactive queries are 
enabled) and thus still end up sending redundant bytes to the broker and also 
logging a large kafka message.

With 100s of streams instances, this overhead can grow into tens of megabytes 
easily.  

*Proposal :*

Extend the group assignment protocol to be able to support passing of an 
additional byte[], which can now contain the HostInfo -> 
partitions/partitionsByHost data just one time. 

{code}
final class GroupAssignment {
private final Map assignments;

// bytes sent to each consumer from leader
private final byte[] globalUserData
...
}
{code}
 
This can generally be handy to any other application like Streams, that does 
some stateful processing or lightweight cluster management 
 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

[Guozhang Wang]

Hi Boyang, Jason,

If we are going to expose the generation id / group.instance id etc anyways
I think its slightly better to just add a new API on KafkaConsumer
returning the ConsumerGroupMetadata (option 3) than passing it in on an
additional callback of ConsumerRebalanceListener.
It feels easier to leverage, than requiring users to pass in the listener.

Guozhang

On Mon, Aug 12, 2019 at 3:41 PM Boyang Chen 
wrote:

> Thanks Jason, the intuition behind defining a separate callback function is
> that, with KIP-429 we no longer guarantee to call OnPartitionsAssigned() or
> OnPartitionsRevoked() with each rebalance. Our requirement is to be
> up-to-date with group metadata such as generation information, so callback
> like onGroupJoined() would make more sense as it should be invoked after
> every successful rebalance.
>
> Best,
> Boyang
>
> On Mon, Aug 12, 2019 at 2:02 PM Jason Gustafson 
> wrote:
>
> > Hey Boyang,
> >
> > I favor option 4 as well. It's a little more cumbersome than 3 for this
> use
> > case, but it seems like a cleaner separation of concerns. The rebalance
> > listener is already concerned with events affecting the assignment
> > lifecycle and group membership. I think the only thing I'm wondering is
> > whether it should be a separate callback as you've suggested, or if it
> > would make sense to overload `onPartitionsAssigned`. If it's separate,
> > maybe a name like `onGroupJoined` would be clearer?
> >
> > Thanks,
> > Jason
> >
> >
> >
> > On Thu, Aug 8, 2019 at 10:59 PM Boyang Chen 
> > wrote:
> >
> > > Thank you Jason. We had some offline discussion on properly keeping
> group
> > > metadata up to date, and here are some of our options brainstormed:
> > > 1. Let the caller of `sendOffsetsToTransaction(offset, metadata)`
> > maintain
> > > the ever-changing group metadata. This could be done on stream side,
> but
> > > for non-stream EOS the sample code will become complicated as the user
> > > needs to implement the partition assignor interface to get the update
> > from
> > > `onAssignment`
> > >
> > > 2. Get a new API on producer like `refreshGroupMetadata(metadata)`.
> This
> > is
> > > similar to option 1 except that now in the partition assignor callback
> we
> > > could straightly pass in the producer instance, which simplifies the
> > > non-stream EOS, however this new API seems weird to define on producer.
> > >
> > > 3. Make an accessing interface to group metadata, or just expose the
> > group
> > > metadata through a consumer API like `consumer.GroupMetadata()`. This
> is
> > > the old way which avoids the users’ effort to implement partition
> > assignor
> > > directly.
> > >
> > > 4. Expose the group metadata through rebalance listener, which is a
> more
> > > well-known and adopted callback interface. We could do sth like
> > > `onGroupMetadataUpdated(ConsumerGroupMetadata metadata)`
> > >
> > > To simplify the code logic, we believe option 3 & 4 are better
> solutions,
> > > and of which I slightly prefer option 4 as it is the most clean
> solution
> > > with less intrusion to both consumer and producer APIs.
> > >
> > > WDYT?
> > >
> > > Boyang
> > >
> > >
> > >
> > >
> > > On Wed, Aug 7, 2019 at 9:20 AM Jason Gustafson 
> > wrote:
> > >
> > > > Hi Boyang,
> > > >
> > > > > We already persist member.id, instance.id and generation.id in the
> > > > offset
> > > > topic, what extra fields we need to store?
> > > >
> > > > Yeah, you're right. I was a little confused and thought this
> > information
> > > > was needed by the transaction coordinator.
> > > >
> > > > > This should be easily done on the stream side as we have
> > > > StreamsPartitionAssignor to reflect metadata changes upon
> > > #onAssignment(),
> > > > but non-stream user has to code the callback by hand, do you think
> the
> > > > convenience we sacrifice here worth the simplification benefit?
> > > >
> > > > Either way, you need a reference to the consumer. I was mostly just
> > > > thinking it would be better to reduce the integration point to its
> > > minimum.
> > > > Have you thought through the implications of needing to keep around a
> > > > reference to the consumer in the producer? What if it gets closed? It
> > > seems
> > > > better not to have to think about these cases.
> > > >
> > > > -Jason
> > > >
> > > > On Tue, Aug 6, 2019 at 9:53 PM Boyang Chen <
> reluctanthero...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thank you for the suggestions Jason. And a side note for Guozhang,
> I
> > > > > updated the KIP to reflect the dependency on 447.
> > > > >
> > > > > On Tue, Aug 6, 2019 at 11:35 AM Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hi Boyang, thanks for the updates. I have a few more comments:
> > > > > >
> > > > > > 1. We are adding some new fields to TxnOffsetCommit to support
> > > > > group-based
> > > > > > fencing. Do we need these fields to be persisted in the offsets
> > topic
> > > > to
> > > > > > ensure that the fencing still works 

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

[Cyrus Vafadari]

I am excited to see this implemented +1 nonbinding

On Tue, Aug 13, 2019 at 2:01 PM Chris Egerton  wrote:

> Nice stuff, Arjun! +1 (non-binding)
>
> On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish 
> wrote:
>
> > Hey everyone,
> >
> > I'd like to start a vote for KIP-495 (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
> > ).
> > This change will make Connect easier to debug in production environment.
> >
> > Based on the discussion, I updated the KIP to reflect how Connect will
> use
> > the changes to the log4j controller to initialize its mBean.
> >
> > For your reference, this is the discussion thread
> > https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html
> >
> > Thanks in advance,
> > Arjun
> >
>

Build failed in Jenkins: kafka-2.0-jdk8 #288

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197)

--
[...truncated 894.76 KB...]
kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption PASSED

kafka.admin.DelegationTokenCommandTest > testDelegationTokenRequests STARTED

kafka.admin.DelegationTokenCommandTest > testDelegationTokenRequests PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeMembersOfNonExistingGroup 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeMembersOfNonExistingGroup 
PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeStateOfExistingGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeStateOfExistingGroup PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeExistingGroup PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeStateWithConsumersWithoutAssignedPartitions STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeStateWithConsumersWithoutAssignedPartitions PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeWithMultipleSubActions 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeWithMultipleSubActions 
PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeGroupOffsetsWithShortInitializationTimeout STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeGroupOffsetsWithShortInitializationTimeout PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeOffsetsOfExistingGroupWithNoMembers STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeOffsetsOfExistingGroupWithNoMembers PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeGroupMembersWithShortInitializationTimeout STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeGroupMembersWithShortInitializationTimeout PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeMembersOfExistingGroup 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeMembersOfExistingGroup 
PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeOffsetsOfExistingGroup 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeOffsetsOfExistingGroup 
PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeOffsetsWithConsumersWithoutAssignedPartitions STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeOffsetsWithConsumersWithoutAssignedPartitions PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithMultiPartitionTopicAndMultipleConsumers STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithMultiPartitionTopicAndMultipleConsumers PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeSimpleConsumerGroup STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeSimpleConsumerGroup PASSED

kafka.admin.DescribeConsumerGroupTest > testDescribeStateOfNonExistingGroup 
STARTED

kafka.admin.DescribeConsumerGroupTest > testDescribeStateOfNonExistingGroup 
PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeMembersWithConsumersWithoutAssignedPartitions STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeMembersWithConsumersWithoutAssignedPartitions PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeMembersOfExistingGroupWithNoMembers STARTED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeMembersOfExistingGroupWithNoMembers PASSED

kafka.admin.DescribeConsumerGroupTest > 
testDescribeWithUnrecognizedNewConsumerOption STARTED

kafka.admin.DescribeConsumerGroupTest > 

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

[Chris Egerton]

Nice stuff, Arjun! +1 (non-binding)

On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish  wrote:

> Hey everyone,
>
> I'd like to start a vote for KIP-495 (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
> ).
> This change will make Connect easier to debug in production environment.
>
> Based on the discussion, I updated the KIP to reflect how Connect will use
> the changes to the log4j controller to initialize its mBean.
>
> For your reference, this is the discussion thread
> https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html
>
> Thanks in advance,
> Arjun
>

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

[Guozhang Wang]

+1 (binding). This is a great KIP, thanks Jason!

Regarding the naming of the zkVersion, I'm actually fine to name it more
generally and leave a note that at the moment its value is defined as the
zk version.


Guozhang


On Mon, Aug 12, 2019 at 2:22 PM Jason Gustafson  wrote:

> Hi Viktor,
>
> I originally named the field `CurrentVersion`. I didn't have 'Zk' in the
> name in anticipation of KIP-500. I thought about it and decided it makes
> sense to keep naming consistent with other APIs. Even if KIP-500 passes,
> there will be some time during which it only refers to the zk version.
> Eventually we'll have to decide whether it makes sense to change the name
> or just introduce a new field.
>
> Thanks,
> Jason
>
> On Fri, Aug 9, 2019 at 9:19 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hey Jason,
> >
> > +1 from me too.
> > One note though: since it's a new protocol we could perhaps rename
> > CurrentZkVersion to something like "IsrEpoch" or "IsrVersion". I think
> > that'd reflect its purpose better.
> >
> > Best,
> > Viktor
> >
> > On Wed, Aug 7, 2019 at 8:37 PM Jason Gustafson 
> wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a vote on KIP-497:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-497%3A+Add+inter-broker+API+to+alter+ISR
> > > .
> > > +1
> > > from me.
> > >
> > > -Jason
> > >
> >
>


-- 
-- Guozhang

[VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

[Arjun Satish]

Hey everyone,

I'd like to start a vote for KIP-495 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect).
This change will make Connect easier to debug in production environment.

Based on the discussion, I updated the KIP to reflect how Connect will use
the changes to the log4j controller to initialize its mBean.

For your reference, this is the discussion thread
https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html

Thanks in advance,
Arjun

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Guozhang Wang]

Hi Mickael,

Thanks for the KIP!

Just some minor comments.

1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be
"AlterOffsetsOptions".

2. I'd suggest we change the future structure of "AlterOffsetsResult" to

*KafkaFuture>>*

This is because we will have a hierarchy of two-layers of errors since we
need to find out the group coordinator first and then issue the commit
offset request (see e.g. the ListConsumerGroupOffsetsResult which exclude
partitions that have errors, or the DeleteMembersResult as part of KIP-345).

If the discover-coordinator returns non-triable error, we would set it on
the first layer of the KafkaFuture, and the per-partition error would be
set on the second layer of the KafkaFuture.


Guozhang


On Tue, Aug 13, 2019 at 9:36 AM Colin McCabe  wrote:

> Hi Mickael,
>
> Considering that KIP-496, which adds a way of deleting consumer offsets
> from AdminClient, looks like it is going to get in, this seems like
> functionality we should definitely have.
>
> For alterConsumerGroupOffsets, is the intention to ignore partitions that
> are not specified in the map?  If so, we should specify that in the JavaDoc.
>
> isolationLevel seems like it should be an enum rather than a string.  The
> existing enum is in org.apache.kafka.common.requests, so we should probably
> create a new one which is public in org.apache.kafka.clients.admin.
>
> best,
> Colin
>
>
> On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> > Bumping this thread once again
> >
> > Ismael, have I answered your questions?
> > While this has received a few non-binding +1s, no committers have
> > voted yet. If you have concerns or questions, please let me know.
> >
> > Thanks
> >
> > On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
> >  wrote:
> > >
> > > Bumping this thread as it's been a couple of weeks.
> > >
> > > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison <
> mickael.mai...@gmail.com> wrote:
> > > >
> > > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > > - Having the reset functionality in the AdminClient:
> > > > The fact we have a command line tool illustrate that this operation
> is
> > > > relatively common. I seems valuable to be able to perform this
> > > > operation directly via a proper API in addition of the CLI tool.
> > > >
> > > > - Sending an OffsetCommit directly instead of relying on
> KafkaConsumer:
> > > > The KafkaConsumer requires a lot of stuff to commit offsets. Its
> group
> > > > cannot change so you need to start a new Consumer every time, that
> > > > creates new connections and overal sends more requests. Also there
> are
> > > > already  a bunch of AdminClient APIs that have logic very close to
> > > > what needs to be done to send a commit request, keeping the code
> small
> > > > and consistent.
> > > >
> > > > I've updated the KIP with these details and moved the 2nd part to
> > > > "Proposed changes" as it's more an implementation detail.
> > > >
> > > > I hope this answers your question
> > > >
> > > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma 
> wrote:
> > > > >
> > > > > The KIP doesn't discuss the option of using KafkaConsumer directly
> as far
> > > > > as I can tell. We have tried to avoid having the same
> functionality in
> > > > > multiple clients so it would be good to explain why this is
> necessary here
> > > > > (not saying it isn't).
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison <
> mickael.mai...@gmail.com
> > > > > wrote:
> > > > >
> > > > > > Thanks Ryanne for the feedback, all suggestions sounded good,
> I've
> > > > > > updated the KIP accordingly.
> > > > > >
> > > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan <
> ryannedo...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > But I suggest:
> > > > > > >
> > > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > > >
> > > > > > > - add to the motivation section why this is better than
> constructing a
> > > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > > >
> > > > > > > - add some rejected alternatives.
> > > > > > >
> > > > > > > Ryanne
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee  wrote:
> > > > > > >
> > > > > > > > We have +4 non-binding for this vote. Is there any committer
> who is
> > > > > > > > interested in this issue?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dongjin
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > > andrew_schofi...@live.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > > >
> > > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" <
> eno.there...@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > +1 (non binding). Thanks.
> > > > > > > > >
> > > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > 

Build failed in Jenkins: kafka-2.2-jdk8 #159

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197)

--
[...truncated 2.73 MB...]
kafka.zk.KafkaZkClientTest > testLogDirGetters STARTED

kafka.zk.KafkaZkClientTest > testLogDirGetters PASSED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment STARTED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion PASSED

kafka.zk.KafkaZkClientTest > testGetChildren STARTED

kafka.zk.KafkaZkClientTest > testGetChildren PASSED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset STARTED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset PASSED

kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

kafka.zk.KafkaZkClientTest > testClusterIdMethods PASSED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr STARTED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr PASSED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testCreateRecursive STARTED

kafka.zk.KafkaZkClientTest > testCreateRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData STARTED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods PASSED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testAclManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testAclManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods STARTED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateLogDir STARTED

kafka.zk.KafkaZkClientTest > testPropagateLogDir PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndStat STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndStat PASSED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress STARTED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress PASSED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths STARTED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters PASSED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetLogConfigs STARTED

kafka.zk.KafkaZkClientTest > testGetLogConfigs PASSED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods STARTED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods PASSED

kafka.zk.KafkaZkClientTest > testAclMethods STARTED

kafka.zk.KafkaZkClientTest > testAclMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath STARTED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath PASSED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath STARTED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode PASSED

kafka.zk.KafkaZkClientTest > testDeletePath STARTED

kafka.zk.KafkaZkClientTest > testDeletePath PASSED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods STARTED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions STARTED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions PASSED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath STARTED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath PASSED

kafka.zk.KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck 
STARTED

kafka.zk.KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck 
PASSED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods PASSED

Re: [VOTE] KIP-503: deleted topics metric

[Stanislav Kozlovski]

+1 (non-binding)

Thanks for the simple but very useful KIP!
Best,
Stanislav

On Tue, Aug 13, 2019 at 8:32 PM Harsha Chintalapani  wrote:

> +1 (binding)
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 13, 2019 at 12:08 PM, David Arthur 
> wrote:
>
> > Hello all,
> >
> > I'd like to start the vote on KIP-503
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
> >
> > Thanks!
> > David
> >
>


-- 
Best,
Stanislav

Build failed in Jenkins: kafka-trunk-jdk8 #3848

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197)

[rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread

--
[...truncated 2.59 MB...]

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor should create a Consumed with Serdes and timestampExtractor 
STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor should create a Consumed with Serdes and timestampExtractor 
PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
resetPolicy should create a Consumed with Serdes and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
resetPolicy should create a Consumed with Serdes and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilWindowCloses should produce the correct suppression STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilWindowCloses should produce the correct suppression PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilTimeLimit should produce the correct suppression STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilTimeLimit should produce the correct suppression PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxRecords 
should produce the correct buffer config STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxRecords 
should produce the correct buffer config PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxBytes 
should produce the correct buffer config STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxBytes 
should produce the correct buffer config PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.unbounded 
should produce the correct buffer config STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.unbounded 
should produce the correct buffer config PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig should 
support very long chains of factory methods STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig should 
support very long chains of factory methods PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialized 
should create a Materialized with Serdes STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialized 
should create a Materialized with Serdes PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a store name should create a Materialized with Serdes and a store name 
STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a store name should create a Materialized with Serdes and a store name 
PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a window store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a window store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a key value store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a key value store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a session store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a session store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced 

Re: [VOTE] KIP-503: deleted topics metric

[Harsha Chintalapani]

+1 (binding)

Thanks,
Harsha


On Tue, Aug 13, 2019 at 12:08 PM, David Arthur 
wrote:

> Hello all,
>
> I'd like to start the vote on KIP-503
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
>
> Thanks!
> David
>

[VOTE] KIP-503: deleted topics metric

[David Arthur]

Hello all,

I'd like to start the vote on KIP-503
https://cwiki.apache.org/confluence/display/KAFKA/KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion

Thanks!
David

Re: [DISCUSS] KIP-503: deleted topics metric

[David Arthur]

Stan, I think that makes sense. I'll update the KIP and start the vote
shortly.

On Thu, Aug 8, 2019 at 12:54 PM Stanislav Kozlovski 
wrote:

> What do people think if we exposed:
> * eligible topics/replicas pending delete
> * ineligible topics/replicas pending delete
>
> On Thu, Aug 8, 2019 at 5:16 PM David Arthur  wrote:
>
> > It looks like topicsIneligibleForDeletion is a subset of
> topicsToBeDeleted
> > in the controller.
> >
> > On Thu, Aug 8, 2019 at 11:16 AM Stanislav Kozlovski <
> > stanis...@confluent.io>
> > wrote:
> >
> > > ineligible replicas/topics are not included in the pending metrics,
> > right?
> > > If so, sounds good to me.
> > >
> > > On Thu, Aug 8, 2019 at 4:12 PM David Arthur  wrote:
> > >
> > > > Yes I think exposing ineligible topics would be useful as well. The
> > > > controller also tracks this ineligible state for replicas. Would that
> > be
> > > > useful to expose as well?
> > > >
> > > > In that case, we'd be up to four new metrics:
> > > > * topics pending delete
> > > > * replicas pending delete
> > > > * ineligible topics
> > > > * ineligible replicas
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Wed, Aug 7, 2019 at 5:16 PM Jason Gustafson 
> > > wrote:
> > > >
> > > > > Thanks for the KIP. This is useful. The controller also maintains a
> > set
> > > > for
> > > > > topics which are awaiting deletion, but currently ineligible. A
> topic
> > > > which
> > > > > is undergoing reassignment, for example, is ineligible for
> deletion.
> > > > Would
> > > > > it make sense to have a metric for this as well?
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Aug 7, 2019 at 1:52 PM David Arthur 
> > wrote:
> > > > >
> > > > > > Updated the KIP with a count of replicas awaiting deletion.
> > > > > >
> > > > > > On Wed, Aug 7, 2019 at 9:37 AM David Arthur 
> > > wrote:
> > > > > >
> > > > > > > Thanks for the feedback, Stan. That's a good point about the
> > > > partition
> > > > > > > count -- I'll poke around and see if I can surface this value
> in
> > > the
> > > > > > > Controller.
> > > > > > >
> > > > > > > On Tue, Aug 6, 2019 at 8:13 AM Stanislav Kozlovski <
> > > > > > stanis...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks for the KIP David,
> > > > > > >>
> > > > > > >> As you mentioned in the KIP - "when a large number of topics
> > > > > > (partitions,
> > > > > > >> really) are deleted at once, it can take significant time for
> > the
> > > > > > >> Controller to process everything.
> > > > > > >> In that sense, does it make sense to have the metric expose
> the
> > > > number
> > > > > > of
> > > > > > >> partitions that are pending deletion, as opposed to topics?
> > > Perhaps
> > > > > even
> > > > > > >> both?
> > > > > > >> My reasoning is that this metric alone wouldn't say much if we
> > had
> > > > one
> > > > > > >> topic with 1000 partitions versus a topic with 1 partition
> > > > > > >>
> > > > > > >> On Mon, Aug 5, 2019 at 8:19 PM Harsha Chintalapani <
> > > ka...@harsha.io
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Thanks for the KIP.  Its useful metric to have.  LGTM.
> > > > > > >> > -Harsha
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Mon, Aug 05, 2019 at 11:24 AM, David Arthur <
> > > > > > davidart...@apache.org>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hello all, I'd like to start a discussion for
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > >> > >
> > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion
> > > > > > >> > >
> > > > > > >> > > Thanks!
> > > > > > >> > > David
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Best,
> > > > > > >> Stanislav
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > David Arthur
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > David Arthur
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
> >
> > --
> > David Arthur
> >
>
>
> --
> Best,
> Stanislav
>


-- 
David Arthur

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

[David Jacot]

Hi Colin,

Thank you for the KIP! Things are well explained!. It is huge improvement
for the Kafka protocol. I have few comments on the proposal:

1. The interleaved tag/length header sounds like a great optimisation as it
would be shorter on average. The downside, as
you already pointed out, is that it makes the decoding and the specs more
complex. Personally, I would also favour using two
vaints in this particular case to keep things simple.

2. As discussed, I wonder if it would make sense to extend to KIP to also
support optional fields in the Record Header. I think
that it could be interesting to have such capability for common fields
across all the requests or responses (e.g. tracing id).

Regards,
David



On Tue, Aug 13, 2019 at 10:00 AM Jason Gustafson  wrote:

> > Right, I was planning on doing exactly that for all the auto-generated
> RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> better use of time to convert the manual ones to auto gen first (with the
> possible exception of Fetch/Produce, where the ROI may be higher for the
> manual work)
>
> Yeah, that makes sense. Maybe we can include the version bump for all RPCs
> in this KIP, but we can implement it lazily as the protocols are converted.
>
> -Jason
>
> On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe  wrote:
>
> > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote:
> > > Hi Colin,
> > >
> > > Thanks for the KIP! This is a significant improvement. One of my
> personal
> > > interests in this proposal is solving the compatibility problems we
> have
> > > with the internal schemas used to define consumer offsets and
> transaction
> > > metadata. Currently we have to guard schema bumps with the inter-broker
> > > protocol format. Once the format is bumped, there is no way to
> downgrade.
> > > By fixing this, we can potentially begin using the new schemas before
> the
> > > IBP is bumped while still allowing downgrade.
> > >
> > > There are a surprising number of other situations we have encountered
> > this
> > > sort of problem. We have hacked around it in special cases by allowing
> > > nullable fields to the end of the schema, but this is not really an
> > > extensible approach. I'm looking forward to having a better option.
> >
> > Yeah, this problem keeps coming up.
> >
> > >
> > > With that said, I have a couple questions on the proposal:
> > >
> > > 1. For each request API, we need one version bump to begin support for
> > > "flexible versions." Until then, we won't have the option of using
> tagged
> > > fields even if the broker knows how to handle them. Does it make sense
> to
> > > go ahead and do a universal bump of each request API now so that we'll
> > have
> > > this option going forward?
> >
> > Right, I was planning on doing exactly that for all the auto-generated
> > RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> > better use of time to convert the manual ones to auto gen first (with the
> > possible exception of Fetch/Produce, where the ROI may be higher for the
> > manual work)
> >
> > > 2. The alternating length/tag header encoding lets us save a byte in
> the
> > > common case. The downside is that it's a bit more complex to specify.
> It
> > > also has some extra cost if the length exceeds the tag substantially.
> > > Basically we'd have to pad the tag, right? I think I'm wondering if we
> > > should just bite the bullet and use two varints instead.
> >
> > That’s a fair point. It would be shorter on average, but worse for some
> > exceptional cases. Also, the decoding would be more complex, which might
> be
> > a good reason to go for just having two varints. Yeah, let’s simplify.
> >
> > Regards,
> > Colin
> >
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Fri, Aug 9, 2019 at 4:31 PM Colin McCabe 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've made some updates to this KIP. Specifically, I wanted to avoid
> > > > including escape bytes in the serialization format, since it was too
> > > > complex. Also, I think this is a good opportunity to slim down our
> > > > variable length fields.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Thu, Jul 11, 2019, at 20:52, Colin McCabe wrote:
> > > > > On Tue, Jul 9, 2019, at 15:29, Jose Armando Garcia Sancio wrote:
> > > > > > Thanks Colin for the KIP. For my own edification why are we doing
> > this
> > > > > > "Optional fields can have any type, except for an array of
> > > > structures."?
> > > > > > Why can't we have an array of structures?
> > > > >
> > > > > Optional fields are serialized starting with their total length.
> This
> > > > > is straightforward to calculate for primitive fields like INT32,
> (or
> > > > > even an array of INT32), but more difficult to calculate for an
> array
> > > > > of structures. Basically, we'd have to do a two-pass serialization
> > > > > where we first calculate the lengths of everything, and then write
> it
> > > > > out.
> > > > >
> > > > > The nice thing 

Build failed in Jenkins: kafka-2.1-jdk8 #222

[Apache Jenkins Server]

See 


Changes:

[rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197)

--
[...truncated 468.87 KB...]

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames PASSED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners STARTED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners PASSED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet STARTED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet PASSED

kafka.server.KafkaConfigTest > testFromPropsInvalid STARTED

kafka.server.KafkaConfigTest > testFromPropsInvalid PASSED

kafka.server.KafkaConfigTest > testInvalidCompressionType STARTED

kafka.server.KafkaConfigTest > testInvalidCompressionType PASSED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault STARTED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testValidCompressionType STARTED

kafka.server.KafkaConfigTest > testValidCompressionType PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid STARTED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
STARTED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
STARTED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes STARTED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes PASSED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation STARTED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED
ERROR: Could not install GRADLE_4_8_1_HOME
java.lang.NullPointerException

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
STARTED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides STARTED

kafka.server.FetchRequestDownConversionConfigTest > 

[jira] [Created] (KAFKA-8798) SaslOAuthBearerSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-8798:
---

 Summary: 
SaslOAuthBearerSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
 Key: KAFKA-8798
 URL: https://issues.apache.org/jira/browse/KAFKA-8798
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 2.4.0
Reporter: Gwen Shapira


https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6937/testReport/junit/kafka.api/SaslOAuthBearerSslEndToEndAuthorizationTest/testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl/

```
Error Message
org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout 
instead of the expected 1 records
Stacktrace
org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout 
instead of the expected 1 records
at 
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
at 
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
at 
org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
at org.scalatest.Assertions.fail(Assertions.scala:1091)
at org.scalatest.Assertions.fail$(Assertions.scala:1087)
at org.scalatest.Assertions$.fail(Assertions.scala:1389)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:822)
at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:781)
at 
kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1312)
at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1320)
at 
kafka.api.EndToEndAuthorizationTest.consumeRecords(EndToEndAuthorizationTest.scala:522)
at 
kafka.api.EndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl(EndToEndAuthorizationTest.scala:361)
```



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

[Jason Gustafson]

> Right, I was planning on doing exactly that for all the auto-generated
RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
better use of time to convert the manual ones to auto gen first (with the
possible exception of Fetch/Produce, where the ROI may be higher for the
manual work)

Yeah, that makes sense. Maybe we can include the version bump for all RPCs
in this KIP, but we can implement it lazily as the protocols are converted.

-Jason

On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe  wrote:

> On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote:
> > Hi Colin,
> >
> > Thanks for the KIP! This is a significant improvement. One of my personal
> > interests in this proposal is solving the compatibility problems we have
> > with the internal schemas used to define consumer offsets and transaction
> > metadata. Currently we have to guard schema bumps with the inter-broker
> > protocol format. Once the format is bumped, there is no way to downgrade.
> > By fixing this, we can potentially begin using the new schemas before the
> > IBP is bumped while still allowing downgrade.
> >
> > There are a surprising number of other situations we have encountered
> this
> > sort of problem. We have hacked around it in special cases by allowing
> > nullable fields to the end of the schema, but this is not really an
> > extensible approach. I'm looking forward to having a better option.
>
> Yeah, this problem keeps coming up.
>
> >
> > With that said, I have a couple questions on the proposal:
> >
> > 1. For each request API, we need one version bump to begin support for
> > "flexible versions." Until then, we won't have the option of using tagged
> > fields even if the broker knows how to handle them. Does it make sense to
> > go ahead and do a universal bump of each request API now so that we'll
> have
> > this option going forward?
>
> Right, I was planning on doing exactly that for all the auto-generated
> RPCs. For the manual RPCs, it would be a lot of work. It’s probably a
> better use of time to convert the manual ones to auto gen first (with the
> possible exception of Fetch/Produce, where the ROI may be higher for the
> manual work)
>
> > 2. The alternating length/tag header encoding lets us save a byte in the
> > common case. The downside is that it's a bit more complex to specify. It
> > also has some extra cost if the length exceeds the tag substantially.
> > Basically we'd have to pad the tag, right? I think I'm wondering if we
> > should just bite the bullet and use two varints instead.
>
> That’s a fair point. It would be shorter on average, but worse for some
> exceptional cases. Also, the decoding would be more complex, which might be
> a good reason to go for just having two varints. Yeah, let’s simplify.
>
> Regards,
> Colin
>
> >
> > Thanks,
> > Jason
> >
> >
> > On Fri, Aug 9, 2019 at 4:31 PM Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > I've made some updates to this KIP. Specifically, I wanted to avoid
> > > including escape bytes in the serialization format, since it was too
> > > complex. Also, I think this is a good opportunity to slim down our
> > > variable length fields.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, Jul 11, 2019, at 20:52, Colin McCabe wrote:
> > > > On Tue, Jul 9, 2019, at 15:29, Jose Armando Garcia Sancio wrote:
> > > > > Thanks Colin for the KIP. For my own edification why are we doing
> this
> > > > > "Optional fields can have any type, except for an array of
> > > structures."?
> > > > > Why can't we have an array of structures?
> > > >
> > > > Optional fields are serialized starting with their total length. This
> > > > is straightforward to calculate for primitive fields like INT32, (or
> > > > even an array of INT32), but more difficult to calculate for an array
> > > > of structures. Basically, we'd have to do a two-pass serialization
> > > > where we first calculate the lengths of everything, and then write it
> > > > out.
> > > >
> > > > The nice thing about this KIP is that there's nothing in the protocol
> > > > stopping us from adding support for this feature in the future. We
> > > > wouldn't have to really change the protocol at all to add support.
> But
> > > > we'd have to change a lot of serialization code. Given almost all of
> > > > our use-cases for optional fields are adding an extra field here or
> > > > there, it seems reasonable not to support it for right now.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[Rajini Sivaram]

Thanks David! I have fixed the typo.

Also made a couple of changes to make the context interfaces more generic.
KafkaRequestContext now returns the 16-bit API key as Colin suggested as
well as the friendly name used in metrics which are useful in audit logs.
`Authorizer#start` is now provided a generic `BrokerInfo` interface that
gives cluster id, broker id and endpoint information. The generic interface
can potentially be used in other broker plugins in future and provides
dynamically generated configs like broker id and ports which are currently
not available to plugins unless these configs are statically configured.
Please let me know if there are any concerns.

Regards,

Rajini

On Tue, Aug 13, 2019 at 4:30 PM David Jacot  wrote:

> Hi Rajini,
>
> Thank you for the update! It looks good to me. There is a typo in the
> `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
>
> Regards,
> David
>
> On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram 
> wrote:
>
> > Hi David,
> >
> > Thanks for reviewing the KIP! Since questions about `authorization mode`
> > and `count` have come up multiple times, I have renamed both.
> >
> > 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> > the resource being authorized is referenced within the request.
> >
> > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> > audit logging in the authorizer. The enum values have javadoc which
> > indicate how the authorization result is used in each of the modes to
> > enable authorizers to log audit messages at the right severity level.
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, Aug 12, 2019 at 5:54 PM David Jacot  wrote:
> >
> > > Hi Rajini,
> > >
> > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > questions/suggestions:
> > >
> > > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > > where you want to go with it but it took me a while to figure it out.
> > > Perhaps, we could come up with a better name than `count`?
> > >
> > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > concept,
> > > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> > either
> > > defined or not. Could you elaborate a bit more on that?
> > >
> > > Thanks,
> > > David
> > >
> > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai 
> > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > 3.2 - This makes sense. Thanks for clarifying.
> > > >
> > > > Rest looks fine. Once the implementations are done, it will be more
> > clear
> > > > on the different values RequestType and Mode.
> > > >
> > > > Thanks
> > > >
> > > > Bosco
> > > >
> > > >
> > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" 
> wrote:
> > > >
> > > > Hi Don,
> > > >
> > > > Thanks for the suggestions. A few responses below:
> > > >
> > > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > > discussion on Colin's suggestions regarding this first.
> > > > 3.2 No, I was thinking of some requests that have an old way of
> > > > authorizing
> > > > and a new way where we have retained the old way for backward
> > > > compatibility. One example is Cluster:Create permission to create
> > > > topics.
> > > > We have replaced this with fine-grained topic create access using
> > > > Topic:Create
> > > > for topic patterns. But we still check if user has Cluster:Create
> > > > first. If
> > > > Denied, the deny is ignored and we check Topic:Create. We dont
> want
> > > to
> > > > log
> > > > DENY for Cluster:Create at INFO level for this, since this is
> not a
> > > > mandatory ACL for creating topics. We will get appropriate logs
> > from
> > > > the
> > > > subsequent Topic:Create in this case.
> > > > 3.3 They are not quite the same. FILTER implies that user
> actually
> > > > requested access to and performed some operation on the filtered
> > > > resources.
> > > > LIST_AUTHORZED did not result in any actual access. User simply
> > > wanted
> > > > to
> > > > know what they are allowed to access.
> > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So
> that
> > > is
> > > > different from authorization mode, operation etc.
> > > >
> > > >
> > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> bo...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > Thanks for clarifying. This is very helpful...
> > > > >
> > > > > #1 - On the Ranger side, we should be able to handle multiple
> > > > requests at
> > > > > the same time. I was just not sure how much processing overhead
> > > will
> > > > be
> > > > > there on the Broker side to split and then consolidate the
> > results.
> > > > If it
> > > > > is negligible,  then this is the better way. It will make it
> > future
> > > > proof.
> > > > > #2 -  I agree, having a single "start" call makes it cleaner.
> The
> > > > > 

[jira] [Created] (KAFKA-8797) BufferUnderflowException: Error reading field 'version' from consumer

[Raman Gupta (JIRA)]

Raman Gupta created KAFKA-8797:
--

 Summary: BufferUnderflowException: Error reading field 'version' 
from consumer
 Key: KAFKA-8797
 URL: https://issues.apache.org/jira/browse/KAFKA-8797
 Project: Kafka
  Issue Type: Bug
Reporter: Raman Gupta


Occassionally I get these errors from my 2.3.0 consumers, talking to 2.3.0 
brokers:

{code}
2019-08-08 16:56:47,235 ERROR — 
red.mic.kaf.AbstractKafkaAutoCommitConsumerService: Exception in polling 
thread. Will die for safety.
EXCEPTION: org.apache.kafka.common.protocol.types.SchemaException: Error 
reading field 'version': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:110) 
~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:106)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:262)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:424)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:353)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216) 
~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201) 
~[kafka-clients-2.3.0.jar:?]
at 
com.redock.microservice.kafka.BasicCommitAfterProcessingConsumer.run(BasicCommitAfterProcessingConsumer.kt:51)
 ~[classes/:?]
at 
com.redock.microservice.kafka.AbstractKafkaAutoCommitConsumerService$start$2.invokeSuspend(AbstractKafkaAutoCommitConsumerService.kt:44)
 [classes/:?]
... suppressed 2 lines
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 
[?:?]
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 [?:?]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
]]
{code}

It seems to happen randomly in consumer restart situations. I use static 
consumer groups.




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

[Colin McCabe]

Hi Mickael,

Considering that KIP-496, which adds a way of deleting consumer offsets from 
AdminClient, looks like it is going to get in, this seems like functionality we 
should definitely have.

For alterConsumerGroupOffsets, is the intention to ignore partitions that are 
not specified in the map?  If so, we should specify that in the JavaDoc.

isolationLevel seems like it should be an enum rather than a string.  The 
existing enum is in org.apache.kafka.common.requests, so we should probably 
create a new one which is public in org.apache.kafka.clients.admin.

best,
Colin


On Mon, Mar 25, 2019, at 06:10, Mickael Maison wrote:
> Bumping this thread once again
> 
> Ismael, have I answered your questions?
> While this has received a few non-binding +1s, no committers have
> voted yet. If you have concerns or questions, please let me know.
> 
> Thanks
> 
> On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison
>  wrote:
> >
> > Bumping this thread as it's been a couple of weeks.
> >
> > On Tue, Jan 22, 2019 at 2:26 PM Mickael Maison  
> > wrote:
> > >
> > > Thanks Ismael for the feedback. I think your point has 2 parts:
> > > - Having the reset functionality in the AdminClient:
> > > The fact we have a command line tool illustrate that this operation is
> > > relatively common. I seems valuable to be able to perform this
> > > operation directly via a proper API in addition of the CLI tool.
> > >
> > > - Sending an OffsetCommit directly instead of relying on KafkaConsumer:
> > > The KafkaConsumer requires a lot of stuff to commit offsets. Its group
> > > cannot change so you need to start a new Consumer every time, that
> > > creates new connections and overal sends more requests. Also there are
> > > already  a bunch of AdminClient APIs that have logic very close to
> > > what needs to be done to send a commit request, keeping the code small
> > > and consistent.
> > >
> > > I've updated the KIP with these details and moved the 2nd part to
> > > "Proposed changes" as it's more an implementation detail.
> > >
> > > I hope this answers your question
> > >
> > > On Mon, Jan 21, 2019 at 7:41 PM Ismael Juma  wrote:
> > > >
> > > > The KIP doesn't discuss the option of using KafkaConsumer directly as 
> > > > far
> > > > as I can tell. We have tried to avoid having the same functionality in
> > > > multiple clients so it would be good to explain why this is necessary 
> > > > here
> > > > (not saying it isn't).
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Jan 21, 2019, 10:29 AM Mickael Maison  > > > wrote:
> > > >
> > > > > Thanks Ryanne for the feedback, all suggestions sounded good, I've
> > > > > updated the KIP accordingly.
> > > > >
> > > > > On Mon, Jan 21, 2019 at 3:43 PM Ryanne Dolan 
> > > > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > But I suggest:
> > > > > >
> > > > > > - drop "get" from getOffset, getTimestamp.
> > > > > >
> > > > > > - add to the motivation section why this is better than 
> > > > > > constructing a
> > > > > > KafkaConsumer and using seek(), commit() etc.
> > > > > >
> > > > > > - add some rejected alternatives.
> > > > > >
> > > > > > Ryanne
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 21, 2019, 7:57 AM Dongjin Lee  > > > > >
> > > > > > > We have +4 non-binding for this vote. Is there any committer who 
> > > > > > > is
> > > > > > > interested in this issue?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dongjin
> > > > > > >
> > > > > > > On Mon, Jan 21, 2019 at 10:33 PM Andrew Schofield <
> > > > > > > andrew_schofi...@live.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding). Thanks for the KIP.
> > > > > > > >
> > > > > > > > On 21/01/2019, 12:45, "Eno Thereska" 
> > > > > wrote:
> > > > > > > >
> > > > > > > > +1 (non binding). Thanks.
> > > > > > > >
> > > > > > > > On Mon, Jan 21, 2019 at 12:30 PM Mickael Maison <
> > > > > > > > mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Bumping this thread. Considering this KIP is relatively 
> > > > > > > > straigh
> > > > > > > > > forward, can we get some votes or feedback if you think 
> > > > > > > > it's
> > > > > not?
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > On Tue, Jan 8, 2019 at 5:40 PM Edoardo Comar <
> > > > > edoco...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > > Thanks Mickael!
> > > > > > > > > >
> > > > > > > > > > On Tue, 8 Jan 2019 at 17:39, Patrik Kleindl <
> > > > > pklei...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > Thanks, sounds very helpful
> > > > > > > > > > > Best regards
> > > > > > > > > > > Patrik
> > > > > > > > > > >
> > > > > > > > > > > > Am 08.01.2019 um 18:10 schrieb Mickael Maison <
> > > > > > > > > mickael.mai...@gmail.com
> > > > > > > > > > > >:
> > > 

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

[Colin McCabe]

Hi Jason,

Thanks for the KIP.

Is there ever a desire to delete all the offsets for a given group?  Should the 
protocol and tools support this?

+1 (binding)

best,
Colin


On Mon, Aug 12, 2019, at 10:57, Guozhang Wang wrote:
> +1 (binding).
> 
> Thanks Jason!
> 
> On Wed, Aug 7, 2019 at 11:18 AM Jason Gustafson  wrote:
> 
> > Hi All,
> >
> > I'd like to start a vote on KIP-496:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-496%3A+Administrative+API+to+delete+consumer+offsets
> > .
> > +1
> > from me of course.
> >
> > -Jason
> >
> 
> 
> -- 
> -- Guozhang
>

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

[Colin McCabe]

That is a good point-- we should get KIP-396 voted on.  I will review it today.

best,
Colin


On Tue, Aug 13, 2019, at 05:58, Gabor Somogyi wrote:
> I've had a look on KIP-396 and until now only 1 binding vote arrived. Hope
> others would consider it as a good solution...
> 
> G
> 
> 
> On Tue, Aug 13, 2019 at 11:52 AM Gabor Somogyi 
> wrote:
> 
> > I've had concerns calling AdminClient.listTopics because on big clusters
> > I've seen OOM because of too many TopicPartitions.
> > On the other this problem already exists in the actual implementation
> > because as Colin said Consumer is doing the same on client side. All in all
> > this part is fine.
> >
> > I've checked all the actual use-cases on Spark side which has to be
> > covered and it looks doable.
> >
> >
> > On Tue, Aug 13, 2019 at 6:01 AM Jungtaek Lim  wrote:
> >
> >> So in overall, AdminClient covers the necessary to retrieve up-to-date
> >> topic-partitions, whereas KIP-396 will cover the necessary to retrieve
> >> offset (EARLIEST, LATEST, timestamp) on partition.
> >>
> >> Gabor, could you please add the input if I'm missing something? I'd like
> >> to
> >> double-check on this.
> >>
> >> Assuming I'm not missing something, what would be preferred next action?
> >> Personally I'd keep this as it is until KIP-396 passes the vote (the vote
> >> for KIP-396 opened at January and it still doesn't pass - 7 months - which
> >> worries me a bit if it's going to pass the vote or not), but I also
> >> respect
> >> the lifecycle of KIP in Kafka community.
> >>
> >> On Tue, Aug 13, 2019 at 12:46 PM Jungtaek Lim  wrote:
> >>
> >> >
> >> >
> >> > On Tue, Aug 13, 2019 at 10:01 AM Colin McCabe 
> >> wrote:
> >> >
> >> >> On Mon, Aug 12, 2019, at 14:54, Jungtaek Lim wrote:
> >> >> > Thanks for the feedbacks Colin and Matthias.
> >> >> >
> >> >> > I agree with you regarding getting topics and partitions via
> >> >> AdminClient,
> >> >> > just curious how much the overhead would be. Would it be lighter, or
> >> >> > heavier? We may not want to list topics in regular intervals - in
> >> plan
> >> >> > phase we want to know up-to-date information so that the calculation
> >> >> from
> >> >> > Spark itself makes sense.
> >> >>
> >> >> It would be lighter. The consumer will periodically refresh metadata
> >> for
> >> >> any topic you are subscribed to. AdminClient doesn’t have the concept
> >> of
> >> >> subscriptions, and won’t refresh topic metadata until you request it.
> >> >>
> >> >
> >> > Sounds great! Happy to hear about that.
> >> >
> >> >
> >> >>
> >> >> >
> >> >> > On the other hands I'm not seeing any information regarding offset in
> >> >> > current AdminClient, which is also one of reason we leverage consumer
> >> >> and
> >> >> > call poll(0). Colin, as you mentioned there're KIPs addressing this,
> >> >> could
> >> >> > you refer KIPs so that we can see whether it would work for our case?
> >> >> > Without support of this we cannot replace our usage of consumer/poll
> >> >> with
> >> >> > AdminClient.
> >> >>
> >> >> KIP-396 is the one for listing offsets in AdminClient.
> >> >>
> >> >
> >> > KIP-396 seems to fit to the needs on Spark's purpose to get offset
> >> > information, even for timestamp. Thanks!
> >> > I'd wish there's a way to get a range of (EARLIEST, LATEST) in one call,
> >> > but not a big deal as it just requires two calls.
> >> >
> >> > >
> >> >> > ps. IMHO it seems to be helpful if there's overloaded `listTopics`
> >> which
> >> >> > receives regex same as consumer subscription via pattern. We would
> >> like
> >> >> to
> >> >> > provide same behavior what Kafka is basically providing as a source.
> >> >>
> >> >> We don’t have a regex listTopics at the moment, though we could add
> >> this.
> >> >> Currently, the regex is done on the client side anyway (although we’d
> >> >> really like to change this in the future). So just listing everything
> >> and
> >> >> filtering locally would be the same performance and behavior as the
> >> >> Consumer.
> >> >>
> >> >
> >> > I see. Good to know regex is done on the client side - I've just
> >> searched
> >> > some code and it applies filter for all topics retrieved from metadata
> >> > fetch. Then it would be mostly no difference on this. Thanks for
> >> confirming.
> >> >
> >> >
> >> >>
> >> >> best,
> >> >> Colin
> >> >>
> >> >> >
> >> >> > On Tue, Aug 13, 2019 at 1:03 AM Matthias J. Sax <
> >> matth...@confluent.io>
> >> >> > wrote:
> >> >> >
> >> >> > > Thanks for the details Jungtaek!
> >> >> > >
> >> >> > > I tend to agree with Colin, that using the AdminClient seems to be
> >> the
> >> >> > > better choice.
> >> >> > >
> >> >> > > You can get all topics via `listTopics()` (and you can refresh this
> >> >> > > information on regular intervals) and match any pattern against the
> >> >> list
> >> >> > > of available topics in the driver.
> >> >> > >
> >> >> > > As you use `assignment()` and store offsets in the Spark
> >> checkpoint,
> >> >> it
> >> >> > > seems that using consumer 

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

[David Jacot]

Hi Rajini,

Thank you for the update! It looks good to me. There is a typo in the
`AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.

Regards,
David

On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram 
wrote:

> Hi David,
>
> Thanks for reviewing the KIP! Since questions about `authorization mode`
> and `count` have come up multiple times, I have renamed both.
>
> 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> the resource being authorized is referenced within the request.
>
> 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> audit logging in the authorizer. The enum values have javadoc which
> indicate how the authorization result is used in each of the modes to
> enable authorizers to log audit messages at the right severity level.
>
> Regards,
>
> Rajini
>
> On Mon, Aug 12, 2019 at 5:54 PM David Jacot  wrote:
>
> > Hi Rajini,
> >
> > Thank you for the KIP. Overall, it looks good to me. I have few
> > questions/suggestions:
> >
> > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > where you want to go with it but it took me a while to figure it out.
> > Perhaps, we could come up with a better name than `count`?
> >
> > 2. I had a hard time trying to understand the `AuthorizationMode`
> concept,
> > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> either
> > defined or not. Could you elaborate a bit more on that?
> >
> > Thanks,
> > David
> >
> > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai 
> wrote:
> >
> > > Hi Rajini
> > >
> > > 3.2 - This makes sense. Thanks for clarifying.
> > >
> > > Rest looks fine. Once the implementations are done, it will be more
> clear
> > > on the different values RequestType and Mode.
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > >
> > > On 8/9/19, 5:08 AM, "Rajini Sivaram"  wrote:
> > >
> > > Hi Don,
> > >
> > > Thanks for the suggestions. A few responses below:
> > >
> > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > discussion on Colin's suggestions regarding this first.
> > > 3.2 No, I was thinking of some requests that have an old way of
> > > authorizing
> > > and a new way where we have retained the old way for backward
> > > compatibility. One example is Cluster:Create permission to create
> > > topics.
> > > We have replaced this with fine-grained topic create access using
> > > Topic:Create
> > > for topic patterns. But we still check if user has Cluster:Create
> > > first. If
> > > Denied, the deny is ignored and we check Topic:Create. We dont want
> > to
> > > log
> > > DENY for Cluster:Create at INFO level for this, since this is not a
> > > mandatory ACL for creating topics. We will get appropriate logs
> from
> > > the
> > > subsequent Topic:Create in this case.
> > > 3.3 They are not quite the same. FILTER implies that user actually
> > > requested access to and performed some operation on the filtered
> > > resources.
> > > LIST_AUTHORZED did not result in any actual access. User simply
> > wanted
> > > to
> > > know what they are allowed to access.
> > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that
> > is
> > > different from authorization mode, operation etc.
> > >
> > >
> > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai 
> > > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > Thanks for clarifying. This is very helpful...
> > > >
> > > > #1 - On the Ranger side, we should be able to handle multiple
> > > requests at
> > > > the same time. I was just not sure how much processing overhead
> > will
> > > be
> > > > there on the Broker side to split and then consolidate the
> results.
> > > If it
> > > > is negligible,  then this is the better way. It will make it
> future
> > > proof.
> > > > #2 -  I agree, having a single "start" call makes it cleaner. The
> > > > Authorization implementation will only have to do the
> > initialization
> > > only
> > > > once.
> > > > #3.1 - Thanks for the clarification. I think it makes sense to
> have
> > > this.
> > > > The term "Mode" might not be explicit enough. Essentially it
> seems
> > > you want
> > > > the Authorizer to know the Intent/Purpose of the authorize call
> and
> > > let the
> > > > Authorizer decide what to log as Audit event. Changing the
> > > class/field name
> > > > or giving more documentation will do.
> > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> > > chaining
> > > > multiple Authorizer? If so,  I am not sure whether the Broker
> would
> > > have
> > > > enough information to make that decision. I feel the Authorizer
> > will
> > > be the
> > > > one who would have that knowledge. E.g. in Ranger we have
> explicit
> > > deny,
> > > > which means no matter what, the request should be denied for the
> > > user/group
> > > > or condition. So 

[jira] [Resolved] (KAFKA-8774) Connect REST API exposes plaintext secrets in tasks endpoint if config value contains additional characters

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch resolved KAFKA-8774.
--
   Resolution: Fixed
 Reviewer: Randall Hauch
Fix Version/s: 2.3.1
   2.4.0
   2.2.2
   2.1.2
   2.0.2

[~wicknicks] did a great job identifying the root cause, which involved how the 
AbstractHerder to correctly identifies task configs that contain variables for 
externalized secrets. The original method incorrectly used `matcher.matches()` 
instead of `matcher.find()`. The former method expects the entire string to 
match the regex, whereas the second one can find a pattern anywhere within the 
input string (which fits this use case more correctly).

This is why the problem is only in the tasks endpoint (no other endpoints) when 
connector configs contain externalized secret variables _plus additional 
characters_. If a config value contains only the variable, the secret is not 
exposed on this task endpoint. 

Arjun added unit tests to cover various cases of a config with externalized 
secrets, and updated system tests to cover case where config value contains 
additional characters besides secret that requires regex pattern to be found 
anywhere in the string (as opposed to complete match).

Merged back to the `2.0` branch, which was when [KIP-297 and externalized 
secrets](https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations)
 were introduced.


> Connect REST API exposes plaintext secrets in tasks endpoint if config value 
> contains additional characters
> ---
>
> Key: KAFKA-8774
> URL: https://issues.apache.org/jira/browse/KAFKA-8774
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.3.0
>Reporter: Oleksandr Diachenko
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.0.2, 2.1.2, 2.2.2, 2.4.0, 2.3.1
>
>
> I have configured a Connector to use externalized secrets, and the following 
> endpoint returns secrets in the externalized form: 
> {code:java}
> curl localhost:8083/connectors/foobar|jq
> {code}
> {code:java}
> {
> "name": "foobar",
> "config": {
> "connector.class": "io.confluent.connect.s3.S3SinkConnector",
> ...
> "consumer.override.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\" 
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "admin.override.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\" 
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "consumer.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\" 
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "producer.override.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\" 
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> "producer.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"${file:/some/secret/path/secrets.properties:kafka.api.key}\" 
> password=\"${file:/some/secret/path/secrets.properties:kafka.api.secret}\";",
> ...
> },
> "tasks": [
> { "connector": "foobar", "task": 0 }
> ],
> "type": "sink"
> }{code}
> But another endpoint returns secrets in plain text:
> {code:java}
> curl localhost:8083/connectors/foobar/tasks|jq
> {code}
> {code:java}
> [
>   {
> "id": {
>   "connector": "lcc-kgkpm",
>   "task": 0
> },
> "config": {
>   "connector.class": "io.confluent.connect.s3.S3SinkConnector",
>   ...
>   "errors.log.include.messages": "true",
>   "flush.size": "1000",
>   "consumer.override.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"OOPS\" password=\"SURPRISE\";",
>   "admin.override.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"OOPS\" password=\"SURPRISE\";",
>   "consumer.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"OOPS\" password=\"SURPRISE\";",
>   "producer.override.sasl.jaas.config": 
> "org.apache.kafka.common.security.plain.PlainLoginModule required 
> username=\"OOPS\" password=\"SURPRISE\";",
>   

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

[Dongjin Lee]

Sorry for being late.

It seems like I found a case which requires a method to update Consumer
metadata. In short, kafka-console-consumer.sh is working differently from
2.1.0 for lack of this functionality.

https://issues.apache.org/jira/browse/KAFKA-8789
https://github.com/apache/kafka/pull/7206

Thanks,
Dongjin

On Tue, Aug 13, 2019 at 9:58 PM Gabor Somogyi 
wrote:

> I've had a look on KIP-396 and until now only 1 binding vote arrived. Hope
> others would consider it as a good solution...
>
> G
>
>
> On Tue, Aug 13, 2019 at 11:52 AM Gabor Somogyi 
> wrote:
>
> > I've had concerns calling AdminClient.listTopics because on big clusters
> > I've seen OOM because of too many TopicPartitions.
> > On the other this problem already exists in the actual implementation
> > because as Colin said Consumer is doing the same on client side. All in
> all
> > this part is fine.
> >
> > I've checked all the actual use-cases on Spark side which has to be
> > covered and it looks doable.
> >
> >
> > On Tue, Aug 13, 2019 at 6:01 AM Jungtaek Lim  wrote:
> >
> >> So in overall, AdminClient covers the necessary to retrieve up-to-date
> >> topic-partitions, whereas KIP-396 will cover the necessary to retrieve
> >> offset (EARLIEST, LATEST, timestamp) on partition.
> >>
> >> Gabor, could you please add the input if I'm missing something? I'd like
> >> to
> >> double-check on this.
> >>
> >> Assuming I'm not missing something, what would be preferred next action?
> >> Personally I'd keep this as it is until KIP-396 passes the vote (the
> vote
> >> for KIP-396 opened at January and it still doesn't pass - 7 months -
> which
> >> worries me a bit if it's going to pass the vote or not), but I also
> >> respect
> >> the lifecycle of KIP in Kafka community.
> >>
> >> On Tue, Aug 13, 2019 at 12:46 PM Jungtaek Lim 
> wrote:
> >>
> >> >
> >> >
> >> > On Tue, Aug 13, 2019 at 10:01 AM Colin McCabe 
> >> wrote:
> >> >
> >> >> On Mon, Aug 12, 2019, at 14:54, Jungtaek Lim wrote:
> >> >> > Thanks for the feedbacks Colin and Matthias.
> >> >> >
> >> >> > I agree with you regarding getting topics and partitions via
> >> >> AdminClient,
> >> >> > just curious how much the overhead would be. Would it be lighter,
> or
> >> >> > heavier? We may not want to list topics in regular intervals - in
> >> plan
> >> >> > phase we want to know up-to-date information so that the
> calculation
> >> >> from
> >> >> > Spark itself makes sense.
> >> >>
> >> >> It would be lighter. The consumer will periodically refresh metadata
> >> for
> >> >> any topic you are subscribed to. AdminClient doesn’t have the concept
> >> of
> >> >> subscriptions, and won’t refresh topic metadata until you request it.
> >> >>
> >> >
> >> > Sounds great! Happy to hear about that.
> >> >
> >> >
> >> >>
> >> >> >
> >> >> > On the other hands I'm not seeing any information regarding offset
> in
> >> >> > current AdminClient, which is also one of reason we leverage
> consumer
> >> >> and
> >> >> > call poll(0). Colin, as you mentioned there're KIPs addressing
> this,
> >> >> could
> >> >> > you refer KIPs so that we can see whether it would work for our
> case?
> >> >> > Without support of this we cannot replace our usage of
> consumer/poll
> >> >> with
> >> >> > AdminClient.
> >> >>
> >> >> KIP-396 is the one for listing offsets in AdminClient.
> >> >>
> >> >
> >> > KIP-396 seems to fit to the needs on Spark's purpose to get offset
> >> > information, even for timestamp. Thanks!
> >> > I'd wish there's a way to get a range of (EARLIEST, LATEST) in one
> call,
> >> > but not a big deal as it just requires two calls.
> >> >
> >> > >
> >> >> > ps. IMHO it seems to be helpful if there's overloaded `listTopics`
> >> which
> >> >> > receives regex same as consumer subscription via pattern. We would
> >> like
> >> >> to
> >> >> > provide same behavior what Kafka is basically providing as a
> source.
> >> >>
> >> >> We don’t have a regex listTopics at the moment, though we could add
> >> this.
> >> >> Currently, the regex is done on the client side anyway (although we’d
> >> >> really like to change this in the future). So just listing everything
> >> and
> >> >> filtering locally would be the same performance and behavior as the
> >> >> Consumer.
> >> >>
> >> >
> >> > I see. Good to know regex is done on the client side - I've just
> >> searched
> >> > some code and it applies filter for all topics retrieved from metadata
> >> > fetch. Then it would be mostly no difference on this. Thanks for
> >> confirming.
> >> >
> >> >
> >> >>
> >> >> best,
> >> >> Colin
> >> >>
> >> >> >
> >> >> > On Tue, Aug 13, 2019 at 1:03 AM Matthias J. Sax <
> >> matth...@confluent.io>
> >> >> > wrote:
> >> >> >
> >> >> > > Thanks for the details Jungtaek!
> >> >> > >
> >> >> > > I tend to agree with Colin, that using the AdminClient seems to
> be
> >> the
> >> >> > > better choice.
> >> >> > >
> >> >> > > You can get all topics via `listTopics()` (and you can refresh
> this
> >> >> > > information on regular 

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

[Gabor Somogyi]

I've had a look on KIP-396 and until now only 1 binding vote arrived. Hope
others would consider it as a good solution...

G


On Tue, Aug 13, 2019 at 11:52 AM Gabor Somogyi 
wrote:

> I've had concerns calling AdminClient.listTopics because on big clusters
> I've seen OOM because of too many TopicPartitions.
> On the other this problem already exists in the actual implementation
> because as Colin said Consumer is doing the same on client side. All in all
> this part is fine.
>
> I've checked all the actual use-cases on Spark side which has to be
> covered and it looks doable.
>
>
> On Tue, Aug 13, 2019 at 6:01 AM Jungtaek Lim  wrote:
>
>> So in overall, AdminClient covers the necessary to retrieve up-to-date
>> topic-partitions, whereas KIP-396 will cover the necessary to retrieve
>> offset (EARLIEST, LATEST, timestamp) on partition.
>>
>> Gabor, could you please add the input if I'm missing something? I'd like
>> to
>> double-check on this.
>>
>> Assuming I'm not missing something, what would be preferred next action?
>> Personally I'd keep this as it is until KIP-396 passes the vote (the vote
>> for KIP-396 opened at January and it still doesn't pass - 7 months - which
>> worries me a bit if it's going to pass the vote or not), but I also
>> respect
>> the lifecycle of KIP in Kafka community.
>>
>> On Tue, Aug 13, 2019 at 12:46 PM Jungtaek Lim  wrote:
>>
>> >
>> >
>> > On Tue, Aug 13, 2019 at 10:01 AM Colin McCabe 
>> wrote:
>> >
>> >> On Mon, Aug 12, 2019, at 14:54, Jungtaek Lim wrote:
>> >> > Thanks for the feedbacks Colin and Matthias.
>> >> >
>> >> > I agree with you regarding getting topics and partitions via
>> >> AdminClient,
>> >> > just curious how much the overhead would be. Would it be lighter, or
>> >> > heavier? We may not want to list topics in regular intervals - in
>> plan
>> >> > phase we want to know up-to-date information so that the calculation
>> >> from
>> >> > Spark itself makes sense.
>> >>
>> >> It would be lighter. The consumer will periodically refresh metadata
>> for
>> >> any topic you are subscribed to. AdminClient doesn’t have the concept
>> of
>> >> subscriptions, and won’t refresh topic metadata until you request it.
>> >>
>> >
>> > Sounds great! Happy to hear about that.
>> >
>> >
>> >>
>> >> >
>> >> > On the other hands I'm not seeing any information regarding offset in
>> >> > current AdminClient, which is also one of reason we leverage consumer
>> >> and
>> >> > call poll(0). Colin, as you mentioned there're KIPs addressing this,
>> >> could
>> >> > you refer KIPs so that we can see whether it would work for our case?
>> >> > Without support of this we cannot replace our usage of consumer/poll
>> >> with
>> >> > AdminClient.
>> >>
>> >> KIP-396 is the one for listing offsets in AdminClient.
>> >>
>> >
>> > KIP-396 seems to fit to the needs on Spark's purpose to get offset
>> > information, even for timestamp. Thanks!
>> > I'd wish there's a way to get a range of (EARLIEST, LATEST) in one call,
>> > but not a big deal as it just requires two calls.
>> >
>> > >
>> >> > ps. IMHO it seems to be helpful if there's overloaded `listTopics`
>> which
>> >> > receives regex same as consumer subscription via pattern. We would
>> like
>> >> to
>> >> > provide same behavior what Kafka is basically providing as a source.
>> >>
>> >> We don’t have a regex listTopics at the moment, though we could add
>> this.
>> >> Currently, the regex is done on the client side anyway (although we’d
>> >> really like to change this in the future). So just listing everything
>> and
>> >> filtering locally would be the same performance and behavior as the
>> >> Consumer.
>> >>
>> >
>> > I see. Good to know regex is done on the client side - I've just
>> searched
>> > some code and it applies filter for all topics retrieved from metadata
>> > fetch. Then it would be mostly no difference on this. Thanks for
>> confirming.
>> >
>> >
>> >>
>> >> best,
>> >> Colin
>> >>
>> >> >
>> >> > On Tue, Aug 13, 2019 at 1:03 AM Matthias J. Sax <
>> matth...@confluent.io>
>> >> > wrote:
>> >> >
>> >> > > Thanks for the details Jungtaek!
>> >> > >
>> >> > > I tend to agree with Colin, that using the AdminClient seems to be
>> the
>> >> > > better choice.
>> >> > >
>> >> > > You can get all topics via `listTopics()` (and you can refresh this
>> >> > > information on regular intervals) and match any pattern against the
>> >> list
>> >> > > of available topics in the driver.
>> >> > >
>> >> > > As you use `assignment()` and store offsets in the Spark
>> checkpoint,
>> >> it
>> >> > > seems that using consumer group management is not a good fit for
>> the
>> >> use
>> >> > > case.
>> >> > >
>> >> > >
>> >> > > Thoughts?
>> >> > >
>> >> > >
>> >> > >
>> >> > > -Matthias
>> >> > >
>> >> > > On 8/12/19 8:22 AM, Colin McCabe wrote:
>> >> > > > Hi,
>> >> > > >
>> >> > > > If there’s no need to consume records in the Spark driver, then
>> the
>> >> > > Consumer is probably the wrong thing to use. Instead, Spark should
>> use

Build failed in Jenkins: kafka-trunk-jdk11 #749

[Apache Jenkins Server]

See 


Changes:

[cmccabe] MINOR: Clean up the sticky partitioner code a bit (#7151)

--
[...truncated 6.38 MB...]
org.apache.kafka.streams.TopologyTest > 
multipleSourcesWithSinksShouldHaveDistinctSubtopologies STARTED

org.apache.kafka.streams.TopologyTest > 
multipleSourcesWithSinksShouldHaveDistinctSubtopologies PASSED

org.apache.kafka.streams.TopologyTest > 
kGroupedStreamZeroArgCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
kGroupedStreamZeroArgCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.TopologyTest > 
sessionWindowNamedMaterializedCountShouldPreserveTopologyStructure STARTED

org.apache.kafka.streams.TopologyTest > 
sessionWindowNamedMaterializedCountShouldPreserveTopologyStructure PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldReturnMillisecondsOnValidDuration STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldReturnMillisecondsOnValidDuration PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowArithmeticExceptionForMaxInstant STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowArithmeticExceptionForMaxInstant PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowArithmeticExceptionForMaxDuration STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowArithmeticExceptionForMaxDuration PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldReturnMillisecondsOnValidInstant STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldReturnMillisecondsOnValidInstant PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowNullPointerExceptionForNullInstant STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowNullPointerExceptionForNullInstant PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldContainsNameAndValueInFailMsgPrefix STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldContainsNameAndValueInFailMsgPrefix PASSED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowNullPointerExceptionForNullDuration STARTED

org.apache.kafka.streams.internals.ApiUtilsTest > 
shouldThrowNullPointerExceptionForNullDuration PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterOuter[caching enabled = true] STARTED

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

[Viktor Somogyi-Vass]

Hi Manikumar,

Yea, I just brought up superuser for the sake of simplicity :).
Anyway, your proposition makes sense to me, I'll modify the KIP for this.

The changes summarized:
1. We'll need a new ACL operation as well (say "CreateUsers") to create the
"UserA can create tokens for UserB, UserC" association. This can be used
via the createAcls API of the AdminClient.
2. CreateToken will be a User level operation (instead of a Cluster level
as in previous drafts). So that means any user who wants to create a
delegation token for other users will have to have an ACL set up by a
higher level user to authorize this.
3. DescribeToken will also be a User level operation. In this case tokenT
owned by userB will be described if userA has a Describe ACL on tokenT or
has a DescribeToken ACL on userB. Note that in the latter case userA will
be able to describe all other tokens belonging to userB.

Would this work for you?

Viktor

On Mon, Aug 12, 2019 at 5:45 PM Colin McCabe  wrote:

> +1 for better access control here. In general, impersonating another user
> seems like it’s equivalent to super user access.
>
> Colin
>
> On Mon, Aug 12, 2019, at 05:43, Manikumar wrote:
> > Hi Viktor,
> >
> > As per the KIP, It's not only superuser, any user with required
> permissions
> > (CreateTokens on Cluster Resource), can create the tokens for other
> users.
> > Current proposed permissions defined like, "UserA can create tokens for
> any
> > user".
> > I am thinking, can we change the permissions like "UserA can create
> tokens
> > for UserB, UserC"?
> >
> > Thanks,
> >
> >
> >
> >
> >
> >
> > On Fri, Aug 9, 2019 at 6:39 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hey Manikumar,
> > >
> > > Thanks for the feedback.
> > > I'm not sure I fully grasp the use-case. Would this be a quota? Do we
> say
> > > something like "there can be 10 active delegation tokens at a time
> that is
> > > created by superuserA for other users"?
> > > I think such a feature could be useful to limit the responsibility of
> said
> > > superuser (and blast radius in case of a faulty/malicious superuser)
> and
> > > also to limit potential programming errors. Do you have other use cases
> > > too?
> > >
> > > Thanks,
> > > Viktor
> > >
> > >
> > > On Tue, Aug 6, 2019 at 1:28 PM Manikumar 
> > > wrote:
> > >
> > > > Hi Viktor,
> > > >
> > > > Thanks for taking over this KP.
> > > >
> > > > Current proposed ACL changes allows users to create tokens for any
> user.
> > > > Thinking again about this, admins may want to configure a user to
> > > > impersonate limited number of other users.
> > > > This allows us to configure fine-grained permissions. But this
> requires
> > > a
> > > > new resourceType "User". What do you think?
> > > >
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > >
> > > > On Wed, Jul 31, 2019 at 2:26 PM Viktor Somogyi-Vass <
> > > > viktorsomo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Folks,
> > > > >
> > > > > I'm starting a vote on this.
> > > > >
> > > > > Viktor
> > > > >
> > > > > On Thu, Jun 27, 2019 at 12:02 PM Viktor Somogyi-Vass <
> > > > > viktorsomo...@gmail.com> wrote:
> > > > >
> > > > > > Hi Folks,
> > > > > >
> > > > > > I took over this issue from Manikumar. Recently another
> motivation
> > > have
> > > > > > been raised in Spark for this (SPARK-28173) and I think it'd be
> great
> > > > to
> > > > > > continue this task.
> > > > > > I updated the KIP and will wait for a few days to get some
> feedback
> > > > then
> > > > > > proceed for the vote.
> > > > > >
> > > > > > Thanks,
> > > > > > Viktor
> > > > > >
> > > > > > On Tue, Dec 11, 2018 at 8:29 AM Manikumar <
> manikumar.re...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Harsha,
> > > > > >>
> > > > > >> Thanks for the review.
> > > > > >>
> > > > > >> With this KIP a designated superuser can create tokens without
> > > > requiring
> > > > > >> individual user credentials.
> > > > > >> Any client can authenticate brokers using the created tokens.
> We may
> > > > not
> > > > > >> call this as impersonation,
> > > > > >> since the clients API calls are executing on their own
> authenticated
> > > > > >> connections.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Manikumar
> > > > > >>
> > > > > >> On Fri, Dec 7, 2018 at 11:56 PM Harsha  wrote:
> > > > > >>
> > > > > >> > Hi Mani,
> > > > > >> > Overall KIP looks good to me. Can we call this
> > > > > >> Impersonation
> > > > > >> > support, which is what the KIP is doing?
> > > > > >> > Also instead of using super.uses as the config which
> essentially
> > > > > giving
> > > > > >> > cluster-wide support to the users, we can introduce
> > > > > impersonation.users
> > > > > >> as
> > > > > >> > a config and users listed in the config are allowed to
> impersonate
> > > > > other
> > > > > >> > users.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Harsha
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Dec 7, 2018, at 3:58 AM, Manikumar 

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

[Gabor Somogyi]

I've had concerns calling AdminClient.listTopics because on big clusters
I've seen OOM because of too many TopicPartitions.
On the other this problem already exists in the actual implementation
because as Colin said Consumer is doing the same on client side. All in all
this part is fine.

I've checked all the actual use-cases on Spark side which has to be covered
and it looks doable.


On Tue, Aug 13, 2019 at 6:01 AM Jungtaek Lim  wrote:

> So in overall, AdminClient covers the necessary to retrieve up-to-date
> topic-partitions, whereas KIP-396 will cover the necessary to retrieve
> offset (EARLIEST, LATEST, timestamp) on partition.
>
> Gabor, could you please add the input if I'm missing something? I'd like to
> double-check on this.
>
> Assuming I'm not missing something, what would be preferred next action?
> Personally I'd keep this as it is until KIP-396 passes the vote (the vote
> for KIP-396 opened at January and it still doesn't pass - 7 months - which
> worries me a bit if it's going to pass the vote or not), but I also respect
> the lifecycle of KIP in Kafka community.
>
> On Tue, Aug 13, 2019 at 12:46 PM Jungtaek Lim  wrote:
>
> >
> >
> > On Tue, Aug 13, 2019 at 10:01 AM Colin McCabe 
> wrote:
> >
> >> On Mon, Aug 12, 2019, at 14:54, Jungtaek Lim wrote:
> >> > Thanks for the feedbacks Colin and Matthias.
> >> >
> >> > I agree with you regarding getting topics and partitions via
> >> AdminClient,
> >> > just curious how much the overhead would be. Would it be lighter, or
> >> > heavier? We may not want to list topics in regular intervals - in plan
> >> > phase we want to know up-to-date information so that the calculation
> >> from
> >> > Spark itself makes sense.
> >>
> >> It would be lighter. The consumer will periodically refresh metadata for
> >> any topic you are subscribed to. AdminClient doesn’t have the concept of
> >> subscriptions, and won’t refresh topic metadata until you request it.
> >>
> >
> > Sounds great! Happy to hear about that.
> >
> >
> >>
> >> >
> >> > On the other hands I'm not seeing any information regarding offset in
> >> > current AdminClient, which is also one of reason we leverage consumer
> >> and
> >> > call poll(0). Colin, as you mentioned there're KIPs addressing this,
> >> could
> >> > you refer KIPs so that we can see whether it would work for our case?
> >> > Without support of this we cannot replace our usage of consumer/poll
> >> with
> >> > AdminClient.
> >>
> >> KIP-396 is the one for listing offsets in AdminClient.
> >>
> >
> > KIP-396 seems to fit to the needs on Spark's purpose to get offset
> > information, even for timestamp. Thanks!
> > I'd wish there's a way to get a range of (EARLIEST, LATEST) in one call,
> > but not a big deal as it just requires two calls.
> >
> > >
> >> > ps. IMHO it seems to be helpful if there's overloaded `listTopics`
> which
> >> > receives regex same as consumer subscription via pattern. We would
> like
> >> to
> >> > provide same behavior what Kafka is basically providing as a source.
> >>
> >> We don’t have a regex listTopics at the moment, though we could add
> this.
> >> Currently, the regex is done on the client side anyway (although we’d
> >> really like to change this in the future). So just listing everything
> and
> >> filtering locally would be the same performance and behavior as the
> >> Consumer.
> >>
> >
> > I see. Good to know regex is done on the client side - I've just searched
> > some code and it applies filter for all topics retrieved from metadata
> > fetch. Then it would be mostly no difference on this. Thanks for
> confirming.
> >
> >
> >>
> >> best,
> >> Colin
> >>
> >> >
> >> > On Tue, Aug 13, 2019 at 1:03 AM Matthias J. Sax <
> matth...@confluent.io>
> >> > wrote:
> >> >
> >> > > Thanks for the details Jungtaek!
> >> > >
> >> > > I tend to agree with Colin, that using the AdminClient seems to be
> the
> >> > > better choice.
> >> > >
> >> > > You can get all topics via `listTopics()` (and you can refresh this
> >> > > information on regular intervals) and match any pattern against the
> >> list
> >> > > of available topics in the driver.
> >> > >
> >> > > As you use `assignment()` and store offsets in the Spark checkpoint,
> >> it
> >> > > seems that using consumer group management is not a good fit for the
> >> use
> >> > > case.
> >> > >
> >> > >
> >> > > Thoughts?
> >> > >
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > > On 8/12/19 8:22 AM, Colin McCabe wrote:
> >> > > > Hi,
> >> > > >
> >> > > > If there’s no need to consume records in the Spark driver, then
> the
> >> > > Consumer is probably the wrong thing to use. Instead, Spark should
> use
> >> > > AdminClient to find out what partitions exist and where, manage
> their
> >> > > offsets, and so on. There are some KIPs under discussion now that
> >> would add
> >> > > the necessary APIs for managing offsets.
> >> > > >
> >> > > > Best,
> >> > > > Colin
> >> > > >
> >> > > > On Mon, Aug 12, 2019, at 07:39, Jungtaek Lim wrote:
> >> > > 

[jira] [Created] (KAFKA-8796) A broker joining the cluster should be able to replicate without impacting the cluster

[Marouane RAJI (JIRA)]

Marouane RAJI created KAFKA-8796:


 Summary: A broker joining the cluster should be able to replicate 
without impacting the cluster
 Key: KAFKA-8796
 URL: https://issues.apache.org/jira/browse/KAFKA-8796
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Marouane RAJI
 Attachments: image-2019-08-13-10-26-19-282.png, 
image-2019-08-13-10-28-42-337.png

Hi, 

We run a cluster of 50 brokers, 1.4M msgs/sec at max, on AWS. We were using 
m4.2xlarge. We are now moving to m5.2xlarge. Everytime we replace a broker from 
scratch (EBSs are linked to ec2 instance..), the byte sent on the replaced 
broker increase significantly and that seem to impact the cluster, increasing 
the produce time and fetch time..

This is our configuration per broker :

 

 
{code:java}
broker.id=11
# Socket Server Settings 
#
# The port the socket server listens on
port=9092

advertised.host.name=ec2-xx-xx-xx-xx.eu-west-1.compute.amazonaws.com
# The number of threads handling network requests
num.network.threads=32
# The number of threads doing disk I/O
num.io.threads=16socket server socket.receive.buffer.bytes=1048576 

socket.request.max.bytes=104857600 # The max time a connection can be idle 
connections.max.idle.ms = 6 

num.partitions=2 

default.replication.factor=2 

auto.leader.rebalance.enable=true 

delete.topic.enable=true 

compression.type=producer 

log.message.format.version=0.9.0.1


message.max.bytes=800 
# The minimum age of a log file to be eligible for deletion 
log.retention.hours=48 

log.retention.bytes=30 

log.segment.bytes=268435456 

log.retention.check.interval.ms=6  

log.cleaner.enable=true 

log.cleaner.dedupe.buffer.size=268435456

replica.fetch.max.bytes=8388608 

replica.fetch.wait.max.ms=500 

replica.lag.time.max.ms=1 

num.replica.fetchers = 3 

# Auto creation of topics on the server 
auto.create.topics.enable=true 

controlled.shutdown.enable=true 

inter.broker.protocol.version=0.10.2 

unclean.leader.election.enabled=True
{code}
 

This is what we notice on replication :

I high increase in byte received on the replaced broker

 

!image-2019-08-13-10-26-19-282.png!

!image-2019-08-13-10-28-42-337.png!

You can't see it the graph above but the increase in produce time stayed high 
for 20minutes..

We didn't see anything out of the ordinary in the logs.

Please let us know if there is anything wrong in our config or if it is a 
potential issue that needs fixing with kafka. 

Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

[Tom Bentley]

Hi Mickael,

Sure, that makes sense so I've updated the KIP.

Kind regards,

Tom

On Mon, Aug 12, 2019 at 12:23 PM Mickael Maison 
wrote:

> Hi Tom,
>
> Thanks for following up on this KIP. This is a great improvement that
> will make policies more powerful and at the same time easier to
> manage.
>
> I just have one question:
> In AbstractRequestMetadata.principal() javadoc, it says the principal
> will be "null" for non authenticated session. Can't we just have the
> default Principal for the Session instead of null? It's possible to
> have Principals for PLAINTEXT sessions or use the default ANONYMOUS
> Principal.
>
> Thanks
>
>
> On Mon, Aug 12, 2019 at 10:52 AM Tom Bentley  wrote:
> >
> > Hi folks,
> >
> > As far as I can see the motivation for KIP-201 is still valid, and as far
> > as I'm aware the changes I made to the KIP back in April addressed the
> > previous comments. Since the issue still needs to be addressed I intend
> to
> > start another vote thread in the near future, but before I do I thought
> I'd
> > check whether anyone has any more comments. So please let me know any
> > feedback for this KIP.
> >
> > Many thanks,
> >
> > Tom
> >
> > On Fri, Apr 12, 2019 at 10:46 AM Tom Bentley 
> wrote:
> >
> > > Hi Rajini,
> > >
> > > I've made a number of changes to the KIP.
> > >
> > > 1. I've added RequestedTopicState.requestedConfigs(). This is obviously
> > > unrelated to supporting alter broker, but I think it goes some way to
> > > addressing one of the points Anna made last year.
> > > Anna, wdyt?
> > >
> > > 2. I've added BrokerState, RequestedBrokerState and
> > > BrokerManagementPolicy. These are largely similar to the interfaces for
> > > topic management, but the lifecycle of a BrokerManagementPolicy needs
> to be
> > > different.
> > >
> > > That's because a BrokerManagementPolicy ought to be Configurable with
> the
> > > broker config, but obviously the broker config can change. Because a
> > > cluster-scoped config might be changed via a different broker we need
> to
> > > hook into the Zookeeper change notification on the broker configs to
> > > instantiate a new BrokerManagementPolicy when broker policy changes.
> We'd
> > > need to cope with policy implementation change happening concurrently
> with
> > > policy enforcement.
> > > And technically there's a race here: Sending changes to cluster-scoped
> > > configs to multiple brokers could result in non-deterministic policy
> > > enforcement.
> > >
> > > One way to avoid that would be to require changes to cluster-scoped
> > > configs to be sent to the controller.
> > > This complexity is annoying because it seems likely that many policy
> > > implementations won't _actually_ depend on the broker config.
> > >
> > > Thoughts?
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > > On Wed, Apr 10, 2019 at 9:48 AM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > wrote:
> > >
> > >> Thanks Tom.
> > >>
> > >> Once you have updated the KIP to support broker config updates, it
> may be
> > >> good to start a new vote thread since the other one is quite old and
> > >> perhaps the KIP has changed since then.
> > >>
> > >>
> > >> On Wed, Apr 10, 2019 at 3:58 AM Tom Bentley 
> wrote:
> > >>
> > >> > Hi Rajini,
> > >> >
> > >> > I'd be happy to do that. I'll try to get it done in the next few
> days.
> > >> >
> > >> > Although there's been quite a lot of interest this, the vote thread
> > >> never
> > >> > got any binding +1, so it's been stuck in limbo for a long time. It
> > >> would
> > >> > be great to get this moving again.
> > >> >
> > >> > Kind regards,
> > >> >
> > >> > Tom
> > >> >
> > >> > On Tue, Apr 9, 2019 at 3:04 PM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi Tom,
> > >> > >
> > >> > > Are you planning to extend this KIP to also include dynamic broker
> > >> config
> > >> > > update (currently covered under AlterConfigPolicy)?
> > >> > >
> > >> > > May be worth sending another note to make progress on this KIP
> since
> > >> it
> > >> > has
> > >> > > been around a while and reading through the threads, it looks like
> > >> there
> > >> > > has been a lot of interest in it.
> > >> > >
> > >> > > Thank you,
> > >> > >
> > >> > > Rajini
> > >> > >
> > >> > >
> > >> > > On Wed, Jan 9, 2019 at 11:25 AM Tom Bentley <
> t.j.bent...@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > Hi Anna and Mickael,
> > >> > > >
> > >> > > > Anna, did you have any comments about the points I made?
> > >> > > >
> > >> > > > Mickael, we really need the vote to be passed before there's
> even
> > >> any
> > >> > > work
> > >> > > > to do. With the exception of Ismael, the KIP didn't seem to get
> the
> > >> > > > attention of any of the other committers.
> > >> > > >
> > >> > > > Kind regards,
> > >> > > >
> > >> > > > Tom
> > >> > > >
> > >> > > > On Thu, 13 Dec 2018 at 18:11, Tom Bentley <
> t.j.bent...@gmail.com>
> > >> > wrote:
> > >> > > >
> > >> > > > > Hi Anna,
> > >> > > > >
> > >> > > > > Firstly, let 

UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

[Alexandre Dupriez]

Hello all,

We run into a scenario where we had misconfigured the replication factor
and the minimum in-sync replicas count in such a way that the replication
factor (either default or defined at the topic level) is strictly lower
than the property min.insync.replicas.

We observed broker metrics reporting UnderReplicatedPartitions = 0 and
UnderMinPartitionIsrCount > 0, and the topic’s partitions were unavailable
for producers (with ack=all) and consumers.

Since it seems to be impossible in this scenario to ever reach the number
of in-sync replicas, making partitions permanently unavailable, it could be
worth to prevent this misconfiguration to make its way to the broker, e.g.
a check could be added when a topic is created to ensure the replication
factor is greater than or equals to the minimum number of in-sync replicas.

I may have missed something though. What do you think?

Thank you,
Alexandre

Build failed in Jenkins: kafka-trunk-jdk8 #3847

[Apache Jenkins Server]

See 


Changes:

[cmccabe] MINOR: Clean up the sticky partitioner code a bit (#7151)

--
[...truncated 2.59 MB...]
org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.unbounded 
should produce the correct buffer config PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig should 
support very long chains of factory methods STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig should 
support very long chains of factory methods PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed should 
create a Consumed with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed should 
create a Consumed with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor should create a Consumed with Serdes and timestampExtractor 
STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
timestampExtractor should create a Consumed with Serdes and timestampExtractor 
PASSED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
resetPolicy should create a Consumed with Serdes and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ConsumedTest > Create a Consumed with 
resetPolicy should create a Consumed with Serdes and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced should 
create a Produced with Serdes PASSED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy STARTED

org.apache.kafka.streams.scala.kstream.ProducedTest > Create a Produced with 
timestampExtractor and resetPolicy should create a Consumed with Serdes, 
timestampExtractor and resetPolicy PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped should 
create a Grouped with Serdes PASSED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name STARTED

org.apache.kafka.streams.scala.kstream.GroupedTest > Create a Grouped with 
repartition topic name should create a Grouped with Serdes, and repartition 
topic name PASSED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes STARTED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes PASSED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes and repartition topic name STARTED

org.apache.kafka.streams.scala.kstream.JoinedTest > Create a Joined should 
create a Joined with Serdes and repartition topic name PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should 

[jira] [Created] (KAFKA-8795) Kafka - Segment deleted earlier than expected

[Prashant (JIRA)]

Prashant created KAFKA-8795:
---

 Summary: Kafka - Segment deleted earlier than expected
 Key: KAFKA-8795
 URL: https://issues.apache.org/jira/browse/KAFKA-8795
 Project: Kafka
  Issue Type: Bug
  Components: log cleaner
Affects Versions: 1.0.0
 Environment: production
Reporter: Prashant


Hi 

We are observing that log segments are getting deleted prematurely even before 
largest message timestamp in segment has not yet reached its retention period. 

We are using broker version kafka_2.11-1.0.0. 

Looking at timeindex file , I see that it is getting append with "timestamp: 0" 
entries with  offset= start offset of segment. 

Example : 

File : 00047730.timeindex

File : 00047730.timeindex

timestamp: 1565565117007 offset: 47799

timestamp: 1565565117037 offset: 47846

timestamp: 1565565117087 offset: 47917

...

timestamp: 1565565118742 offset: 50607

*timestamp: 0* offset: 47730

timestamp: 0 offset: 47730

 

Last message published to this segment was at 1565565118742  which is 00:11 BST 
and segment got deleted by log cleaner at 12:12 BST. 

 

 

[12:12:34,143] INFO Rolled new log segment for 'TOPICNAME-0' in 50 ms. 
(kafka.log.Log)
[12:12:34,143] INFO Scheduling log segment 47942 for log TOPICNAME-0 for 
deletion. (kafka.log.Log)
[12:12:34,147] INFO Incrementing log start offset of partition TOPICNAME-0 to 
50749 in dir /home/test/data-5 (kafka.log.Log)
[12:12:34,149] INFO Cleared earliest 0 entries from epoch cache based on passed 
offset 50749 leaving 1 in EpochFile for partition TOPICNAME-0 
(kafka.server.epoch.LeaderEpochFileCache)
[12:13:34,147] INFO Deleting segment 47942 from log TOPICNAME-0. (kafka.log.Log)
[12:13:34,148] INFO Deleting index 
/home/test/data-5/TOPICNAME-0/00047942.index.deleted 
(kafka.log.OffsetIndex)
[12:13:34,149] INFO Deleting index 
/home/test/data-5/TOPICNAME-0/00047942.timeindex.deleted 
(kafka.log.TimeIndex)

 

Please help us understand what is causing this issue and how to fix it. 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KAFKA-8794) Provide Javadoc on DescribeLogDirsResult

[Lee Dongjin (JIRA)]

Lee Dongjin created KAFKA-8794:
--

 Summary: Provide Javadoc on DescribeLogDirsResult
 Key: KAFKA-8794
 URL: https://issues.apache.org/jira/browse/KAFKA-8794
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: Lee Dongjin
Assignee: Lee Dongjin


As of 2.3.0, DescribeLogDirsResult returned by 
AdminClient#describeLogDirs(Collection) is exposing the internal data 
structure, DescribeLogDirsResponse.LogDirInfo. By doing so, its Javadoc 
provides no documentation on it. Its imparity is clear when comparing with 
DescribeReplicaLogDirsResult, returned by 
AdminClient#describeReplicaLogDirs(Collection).

To resolve this, org.apache.kafka.clients.admin.DescribeLogDirsResult should 
provide [LogDirInfo, ReplicaInfo] as its internal class, like 
DescribeReplicaLogDirsResult.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KAFKA-8793) StickyTaskAssignor throws java.lang.ArithmeticException

[Raman Gupta (JIRA)]

Raman Gupta created KAFKA-8793:
--

 Summary: StickyTaskAssignor throws java.lang.ArithmeticException
 Key: KAFKA-8793
 URL: https://issues.apache.org/jira/browse/KAFKA-8793
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.3.0
Reporter: Raman Gupta


Occassionally when starting a streams consumer that uses the static consumer 
group protocol, I get the following error:
{code:java}
2019-08-13 06:06:43,527 ERROR --- [691d2-StreamThread-1] 
org.apa.kaf.str.pro.int.StreamThread : stream-thread 
[prod-cisSegmenter-777489d8-6cc5-48b4-8771-868d873691d2-StreamThread-1] 
Encountered the following er
ror during processing:
EXCEPTION: java.lang.ArithmeticException: / by zero
at 
org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignor.assignActive(StickyTaskAssignor.java:76)
 ~[kafka-streams-2.3.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignor.assign(StickyTaskAssignor.java:52)
 ~[kafka-streams-2.3.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:634)
 ~[kafka-streams-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:424)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:622)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:107)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:544)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:527)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:978)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:958)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:578)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:388)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:294)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:415)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:353)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
 ~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216) 
~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201) 
~[kafka-clients-2.3.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:941)
 ~[kafka-streams-2.3.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:850)
 ~[kafka-streams-2.3.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
 ~[kafka-streams-2.3.0.jar:?]
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
 [kafka-streams-2.3.0.jar:?]
{code}

It seems to happen after a restart of a process containing a stream, and it 
does not happen consistently, however it does happen somewhat regularly.

My Kafka server is 2.3.0, with a patch for KAFKA-8715.



--
This message was sent by