Build failed in Jenkins: kafka-2.3-jdk8 #86

See Changes: [bill] KAFKA-8736: Track size in InMemoryKeyValueStore (#7177) [matthias] KAFKA-8791: RocksDBTimestampedStore should open in regular mode by -- [...truncated 2.2

Build failed in Jenkins: kafka-trunk-jdk11 #751

See Changes: [gwen] KAFKA-8391; Improved the Connect integration tests to make them less [jason] MINOR: Add fetch from follower system test (#7166) [bbejeck] KAFKA-8736: Track size in InMemoryKeyValueStore (#71

[jira] [Resolved] (KAFKA-8792) Default ZK configuration to disable AdminServer

[ https://issues.apache.org/jira/browse/KAFKA-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-8792. - Resolution: Fixed Reviewer: Ismael Juma Fix Version/s: 2.4.0 > Default ZK configu

[jira] [Resolved] (KAFKA-8695) Metrics UnderReplicated and UnderMinIsr are diverging when configuration is inconsistent

[ https://issues.apache.org/jira/browse/KAFKA-8695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Dupriez resolved KAFKA-8695. -- Resolution: Duplicate De-duplicating in favor of  [KAFKA-4680|http://issues.apache.org/

Build failed in Jenkins: kafka-2.1-jdk8 #224

See Changes: [rhauch] KAFKA-8391; Improved the Connect integration tests to make them less -- [...truncated 926.82 KB...] kafka.utils.CoreUtilsTest > testReadInt STARTED ka

Re: UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

Alexandre, You are right that this is a problem. There is a JIRA on this from a while back. https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4680 I don’t think anyone is currently working on it right now. -James Sent from my iPhone > On Aug 13, 2019, at 1:17 AM, Alexandre

Jenkins build is back to normal : kafka-2.0-jdk8 #289

See

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

Hi Colin, Thanks for the KIP. Optional fields and var length encoding support is a great improvement for the protocol. >>Optional fields can have any type, except that they cannot be arrays. Note that the restriction against having tagged arrays is just to simplify serialization. We can relax thi

[jira] [Created] (KAFKA-8800) Flaky Test SaslScramSslEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

Matthias J. Sax created KAFKA-8800: -- Summary: Flaky Test SaslScramSslEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl Key: KAFKA-8800 URL: https://issues.apache.org/jira/browse/KAFK

Build failed in Jenkins: kafka-2.2-jdk8 #160

See Changes: [rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread [rhauch] KAFKA-8391; Improved the Connect integration tests to make them less -- [...

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

Hey Guozhang, thanks for the suggestion. Could you elaborate more on why defining a direct consumer API would be easier? The benefit of reusing consumer rebalance listener is to consolidate the entry point of consumer internal states. Compared with letting consumer generate a deep-copy of metadata

Build failed in Jenkins: kafka-trunk-jdk8 #3849

See Changes: [gwen] KAFKA-8391; Improved the Connect integration tests to make them less [jason] MINOR: Add fetch from follower system test (#7166) -- [...truncated 2.59

Build failed in Jenkins: kafka-2.1-jdk8 #223

See Changes: [rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread -- [...truncated 685.27 KB...] kafka.api.UserQuotaTest > testProducerConsumerOverride

[jira] [Resolved] (KAFKA-7941) Connect KafkaBasedLog work thread terminates when getting offsets fails because broker is unavailable

[ https://issues.apache.org/jira/browse/KAFKA-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-7941. -- Resolution: Fixed Reviewer: Randall Hauch Fix Version/s: 2.3.1

Build failed in Jenkins: kafka-trunk-jdk11 #750

See Changes: [rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197) [rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread -- [...tr

[jira] [Created] (KAFKA-8799) Support ability to pass global user data to consumers during Assignment

Vinoth Chandar created KAFKA-8799: - Summary: Support ability to pass global user data to consumers during Assignment Key: KAFKA-8799 URL: https://issues.apache.org/jira/browse/KAFKA-8799 Project: Kafk

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

Hi Boyang, Jason, If we are going to expose the generation id / group.instance id etc anyways I think its slightly better to just add a new API on KafkaConsumer returning the ConsumerGroupMetadata (option 3) than passing it in on an additional callback of ConsumerRebalanceListener. It feels easier

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

I am excited to see this implemented +1 nonbinding On Tue, Aug 13, 2019 at 2:01 PM Chris Egerton wrote: > Nice stuff, Arjun! +1 (non-binding) > > On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish > wrote: > > > Hey everyone, > > > > I'd like to start a vote for KIP-495 ( > > > > > https://cwiki.apac

Build failed in Jenkins: kafka-2.0-jdk8 #288

See Changes: [rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197) -- [...truncated 894.76 KB...] kafka.admin.ReassignPartitionsCommandArgsTest > shouldNo

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

Nice stuff, Arjun! +1 (non-binding) On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish wrote: > Hey everyone, > > I'd like to start a vote for KIP-495 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect > ). > This change will make Connect easier

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

+1 (binding). This is a great KIP, thanks Jason! Regarding the naming of the zkVersion, I'm actually fine to name it more generally and leave a note that at the moment its value is defined as the zk version. Guozhang On Mon, Aug 12, 2019 at 2:22 PM Jason Gustafson wrote: > Hi Viktor, > > I o

[VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

Hey everyone, I'd like to start a vote for KIP-495 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect). This change will make Connect easier to debug in production environment. Based on the discussion, I updated the KIP to reflect how Connect w

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

Hi Mickael, Thanks for the KIP! Just some minor comments. 1. Java class names are stale, e.g. "CommitOffsetsOptions.java" should be "AlterOffsetsOptions". 2. I'd suggest we change the future structure of "AlterOffsetsResult" to *KafkaFuture>>* This is because we will have a hierarchy of two-l

Build failed in Jenkins: kafka-2.2-jdk8 #159

See Changes: [rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197) -- [...truncated 2.73 MB...] kafka.zk.KafkaZkClientTest > testLogDirGetters STARTED kaf

Re: [VOTE] KIP-503: deleted topics metric

+1 (non-binding) Thanks for the simple but very useful KIP! Best, Stanislav On Tue, Aug 13, 2019 at 8:32 PM Harsha Chintalapani wrote: > +1 (binding) > > Thanks, > Harsha > > > On Tue, Aug 13, 2019 at 12:08 PM, David Arthur > wrote: > > > Hello all, > > > > I'd like to start the vote on KIP-50

Build failed in Jenkins: kafka-trunk-jdk8 #3848

See Changes: [rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197) [rhauch] KAFKA-7941: Catch TimeoutException in KafkaBasedLog worker thread -- [...tr

Re: [VOTE] KIP-503: deleted topics metric

+1 (binding) Thanks, Harsha On Tue, Aug 13, 2019 at 12:08 PM, David Arthur wrote: > Hello all, > > I'd like to start the vote on KIP-503 > https://cwiki.apache.org/confluence/display/KAFKA/ > KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion > > Thanks! > David >

[VOTE] KIP-503: deleted topics metric

Hello all, I'd like to start the vote on KIP-503 https://cwiki.apache.org/confluence/display/KAFKA/KIP-503%3A+Add+metric+for+number+of+topics+marked+for+deletion Thanks! David

Re: [DISCUSS] KIP-503: deleted topics metric

Stan, I think that makes sense. I'll update the KIP and start the vote shortly. On Thu, Aug 8, 2019 at 12:54 PM Stanislav Kozlovski wrote: > What do people think if we exposed: > * eligible topics/replicas pending delete > * ineligible topics/replicas pending delete > > On Thu, Aug 8, 2019 at 5:

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

Hi Colin, Thank you for the KIP! Things are well explained!. It is huge improvement for the Kafka protocol. I have few comments on the proposal: 1. The interleaved tag/length header sounds like a great optimisation as it would be shorter on average. The downside, as you already pointed out, is th

Build failed in Jenkins: kafka-2.1-jdk8 #222

See Changes: [rhauch] KAFKA-8774: Regex can be found anywhere in config value (#7197) -- [...truncated 468.87 KB...] kafka.server.KafkaConfigTest > testListenerAndAdvertised

[jira] [Created] (KAFKA-8798) SaslOAuthBearerSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

Gwen Shapira created KAFKA-8798: --- Summary: SaslOAuthBearerSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl Key: KAFKA-8798 URL: https://issues.apache.org/jira/browse/KAFKA-8798

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

> Right, I was planning on doing exactly that for all the auto-generated RPCs. For the manual RPCs, it would be a lot of work. It’s probably a better use of time to convert the manual ones to auto gen first (with the possible exception of Fetch/Produce, where the ROI may be higher for the manual wo

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

Thanks David! I have fixed the typo. Also made a couple of changes to make the context interfaces more generic. KafkaRequestContext now returns the 16-bit API key as Colin suggested as well as the friendly name used in metrics which are useful in audit logs. `Authorizer#start` is now provided a ge

[jira] [Created] (KAFKA-8797) BufferUnderflowException: Error reading field 'version' from consumer

Raman Gupta created KAFKA-8797: -- Summary: BufferUnderflowException: Error reading field 'version' from consumer Key: KAFKA-8797 URL: https://issues.apache.org/jira/browse/KAFKA-8797 Project: Kafka

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

Hi Mickael, Considering that KIP-496, which adds a way of deleting consumer offsets from AdminClient, looks like it is going to get in, this seems like functionality we should definitely have. For alterConsumerGroupOffsets, is the intention to ignore partitions that are not specified in the ma

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

Hi Jason, Thanks for the KIP. Is there ever a desire to delete all the offsets for a given group? Should the protocol and tools support this? +1 (binding) best, Colin On Mon, Aug 12, 2019, at 10:57, Guozhang Wang wrote: > +1 (binding). > > Thanks Jason! > > On Wed, Aug 7, 2019 at 11:18 AM

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

That is a good point-- we should get KIP-396 voted on. I will review it today. best, Colin On Tue, Aug 13, 2019, at 05:58, Gabor Somogyi wrote: > I've had a look on KIP-396 and until now only 1 binding vote arrived. Hope > others would consider it as a good solution... > > G > > > On Tue, Au

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

Hi Rajini, Thank you for the update! It looks good to me. There is a typo in the `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`. Regards, David On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram wrote: > Hi David, > > Thanks for reviewing the KIP! Since questions about `authoriza

[jira] [Resolved] (KAFKA-8774) Connect REST API exposes plaintext secrets in tasks endpoint if config value contains additional characters

[ https://issues.apache.org/jira/browse/KAFKA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-8774. -- Resolution: Fixed Reviewer: Randall Hauch Fix Version/s: 2.3.1

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

Sorry for being late. It seems like I found a case which requires a method to update Consumer metadata. In short, kafka-console-consumer.sh is working differently from 2.1.0 for lack of this functionality. https://issues.apache.org/jira/browse/KAFKA-8789 https://github.com/apache/kafka/pull/7206

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

I've had a look on KIP-396 and until now only 1 binding vote arrived. Hope others would consider it as a good solution... G On Tue, Aug 13, 2019 at 11:52 AM Gabor Somogyi wrote: > I've had concerns calling AdminClient.listTopics because on big clusters > I've seen OOM because of too many Topic

Build failed in Jenkins: kafka-trunk-jdk11 #749

See Changes: [cmccabe] MINOR: Clean up the sticky partitioner code a bit (#7151) -- [...truncated 6.38 MB...] org.apache.kafka.streams.TopologyTest > multipleSourcesWithS

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

Hi Manikumar, Yea, I just brought up superuser for the sake of simplicity :). Anyway, your proposition makes sense to me, I'll modify the KIP for this. The changes summarized: 1. We'll need a new ACL operation as well (say "CreateUsers") to create the "UserA can create tokens for UserB, UserC" as

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

I've had concerns calling AdminClient.listTopics because on big clusters I've seen OOM because of too many TopicPartitions. On the other this problem already exists in the actual implementation because as Colin said Consumer is doing the same on client side. All in all this part is fine. I've chec

[jira] [Created] (KAFKA-8796) A broker joining the cluster should be able to replicate without impacting the cluster

Marouane RAJI created KAFKA-8796: Summary: A broker joining the cluster should be able to replicate without impacting the cluster Key: KAFKA-8796 URL: https://issues.apache.org/jira/browse/KAFKA-8796

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Mickael, Sure, that makes sense so I've updated the KIP. Kind regards, Tom On Mon, Aug 12, 2019 at 12:23 PM Mickael Maison wrote: > Hi Tom, > > Thanks for following up on this KIP. This is a great improvement that > will make policies more powerful and at the same time easier to > manage.

UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

Hello all, We run into a scenario where we had misconfigured the replication factor and the minimum in-sync replicas count in such a way that the replication factor (either default or defined at the topic level) is strictly lower than the property min.insync.replicas. We observed broker metrics r

Build failed in Jenkins: kafka-trunk-jdk8 #3847

See Changes: [cmccabe] MINOR: Clean up the sticky partitioner code a bit (#7151) -- [...truncated 2.59 MB...] org.apache.kafka.streams.scala.kstream.SuppressedTest > Buffe

[jira] [Created] (KAFKA-8795) Kafka - Segment deleted earlier than expected

Prashant created KAFKA-8795: --- Summary: Kafka - Segment deleted earlier than expected Key: KAFKA-8795 URL: https://issues.apache.org/jira/browse/KAFKA-8795 Project: Kafka Issue Type: Bug C