Build failed in Jenkins: kafka-trunk-jdk8 #2861

See Changes: [github] MINOR: Add Timer to simplify timeout bookkeeping and use it in the -- [...truncated 2.48 MB...] org.apache.kafka.streams.TopologyTest > multipleSour

[jira] [Resolved] (KAFKA-6662) Consumer use offsetsForTimes() get offset return None.

[ https://issues.apache.org/jira/browse/KAFKA-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Wang resolved KAFKA-6662. -- Resolution: Not A Problem > Consumer use offsetsForTimes() get offset return None. > --

Build failed in Jenkins: kafka-trunk-jdk10 #366

See Changes: [github] MINOR: Add Timer to simplify timeout bookkeeping and use it in the -- [...truncated 1.54 MB...] kafka.coordinator.transaction.TransactionCoordinatorT

[jira] [Created] (KAFKA-7245) Deprecate WindowStore#put(key, value)

Matthias J. Sax created KAFKA-7245: -- Summary: Deprecate WindowStore#put(key, value) Key: KAFKA-7245 URL: https://issues.apache.org/jira/browse/KAFKA-7245 Project: Kafka Issue Type: Improveme

Build failed in Jenkins: kafka-trunk-jdk8 #2860

See Changes: [mjsax] MINOR: Require final variables in Streams (#5452) -- [...truncated 2.47 MB...] org.apache.kafka.streams.TopologyTest > multipleSourcesWithProcessors

Build failed in Jenkins: kafka-trunk-jdk10 #365

See Changes: [mjsax] MINOR: Require final variables in Streams (#5452) -- [...truncated 1.98 MB...] org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce ST

Re: [DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Guozhang: I agree. Probably note this on your KIP. Thanks On Fri, Aug 3, 2018 at 6:08 PM Guozhang Wang wrote: > Hello Ted, > > I think dynamic configuration itself would worth an independent KIP, if you > meant to allow users changing the config on-the-fly while not bouncing the > instance. >

Re: [Vote] KIP-321: Update TopologyDescription to better represent Source and Sink Nodes

KIP 321 has passed. Here is the vote break down: Binding: - Matthias J. Sax - Guozhang Wong - Ewen Cheslack-Postava Non-Binding: - Ted Yu - Bill Bejeck - Damian Guy Thanks to all those who voted and provided feedback! Best, Nishanth Pradeep On Thu, Aug 2, 2018 at 12:42 PM

Re: [DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Hello Ted, I think dynamic configuration itself would worth an independent KIP, if you meant to allow users changing the config on-the-fly while not bouncing the instance. Guozhang On Fri, Aug 3, 2018 at 3:33 PM, Ted Yu wrote: > Guozhang: > > Do you plan to support dynamic config for the new

Re: [DISCUSS] KIP-349 Priorities for Source Topics

Thanks for the KIP. One general question: The Jira is marked as "stream", so I am wondering what the intended scope the KIP is because, it suggests a new consumer API only. Can you clarify? -Matthias On 8/3/18 1:13 PM, Afshartous, Nick wrote: > > Hi All, > > This message is to initiate discuss

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hi, Jason, Thanks for the updated KIP. Well thought-through. Just a few minor comments below. 50. For seek(TopicPartition partition, OffsetAndMetadata offset), I guess under the cover, it will make OffsetsForLeaderEpoch request to determine if the seeked offset is still valid before fetching? If

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

Thanks Manikumar! +1 On Fri, Aug 3, 2018 at 4:33 PM, Jason Gustafson wrote: > +1 Thanks Manikumar! > > On Fri, Aug 3, 2018 at 4:24 PM, Colin McCabe wrote: > > > +1 (non-binding) > > > > regards, > > Colin > > > > On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > > > Hi Manikumar, > > > >

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

+1 Thanks Manikumar! On Fri, Aug 3, 2018 at 4:24 PM, Colin McCabe wrote: > +1 (non-binding) > > regards, > Colin > > On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > > Hi Manikumar, > > > > +1 (binding) > > > > Thanks for the KIP! > > > > On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > >

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

+1 (non-binding) regards, Colin On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > Hi Manikumar, > > +1 (binding) > > Thanks for the KIP! > > On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > > > +1 > > > > On Thu, Aug 2, 2018 at 7:33 PM Manikumar > > wrote: > > > > > Hi All, > > > > > >

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

Hi Stanislav, Thanks for the KIP. As as user, "FaultyRecordException" seems a little vague. What's faulty about it? Is it too big? Does it not match the schema of the other records? Was it not found? Of course, we know that the specific problem is with [de]serializing the record, not with

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Here's hoping someone has time to pick up KIP-14 again: https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization. The inconsistencies are so irritating. -Jason On Fri, Aug 3, 2018 at 10:27 AM, Rajini Sivaram wrote: > Hi Dong, > > I don't have a preference for the option

Re: [DISCUSS] KIP-350: Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it

Thanks for the KIP, Ratish. We should probably specify that it is an error when --broker is specified with operations other than --list, right? best, Colin On Wed, Aug 1, 2018, at 21:28, Ratish Ravindran wrote: > Hi, > > I would like to open a discussion thread on KIP-350: > > https://cwiki.

Re: [DISCUSS] KIP-348 Eliminate null from SourceTask#poll()

"No changes to public interface" doesn't seem accurate here. SourceTask#poll is a public interface, right? Kafka connectors that are out-of-tree would certainly break if we disallowed returning null from this method. However, reading the KIP more closely, it seems like both null and the empty

Re: [VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

+1 Thanks Stanislav! On Fri, Aug 3, 2018 at 2:13 AM, Stanislav Kozlovski wrote: > Thank you, Jun. Noted in the KIP. > > On Fri, Aug 3, 2018 at 2:28 AM Jun Rao wrote: > > > Hi, Stanislav, > > > > Thanks for the KIP. +1 > > > > Just one minor comment. Since the JWT token supports customizable cla

Re: [DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Guozhang: Do you plan to support dynamic config for the new config entry ? Cheers On Fri, Aug 3, 2018 at 2:00 PM Guozhang Wang wrote: > Hello all, > > I would like to kick off a discussion on the following KIP, to allow users > control when a task can be processed based on its buffered records

Jenkins build is back to normal : kafka-2.0-jdk8 #102

See

Jenkins build is back to normal : kafka-trunk-jdk8 #2859

See

Moving build job Was: Build failed in Jenkins: kafka-trunk-jdk10 #364

Should the build job run on nodes with the {{Hadoop}} label ? Based on recent builds@a.o discussion about how small the disk capacity is on the general {{ubuntu}} label, I think we should try {{Hadoop}} labeled machines. Cheers On Fri, Aug 3, 2018 at 3:08 PM Apache Jenkins Server < jenk...@build

[jira] [Resolved] (KAFKA-5153) KAFKA Cluster : 0.10.2.0 : Servers Getting disconnected : Service Impacting

[ https://issues.apache.org/jira/browse/KAFKA-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang resolved KAFKA-5153. --- Resolution: Information Provided Fix Version/s: 0.11.0.1 Upgrading fixed the problem based on

Build failed in Jenkins: kafka-trunk-jdk10 #364

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H31 (ubuntu xenial) in workspace

Re: KIP-352: Distinguish URPs caused by reassignment

@Dong Ugh, you are right. This is indeed trickier than I imagined. You could argue that the problem here is that there is no single source of truth for the expected replication factor. While a reassignment is in progress, the reassignment itself has the expected replica set. Otherwise, it is store

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

@Boyang, * Yes registry id should be generated via a config, similar to what you proposed in the KIP currently for member id. And "even if multiple consumers are using the same member id" (I think you meant registry id here?), we can fence against them. BTW I'm not married (but cannot speak for J

Build failed in Jenkins: kafka-trunk-jdk10 #363

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H31 (ubuntu xenial) in workspace

[DISCUSS] KIP-353: Allow Users to Configure Kafka Streams Timestamp Synchronization

Hello all, I would like to kick off a discussion on the following KIP, to allow users control when a task can be processed based on its buffered records, and how the stream time of a task be advanced. https://cwiki.apache.org/confluence/display/KAFKA/KIP-353%3A+Improve+Kafka+Streams+Timestamp+Syn

[jira] [Created] (KAFKA-7244) Add note about memory map kernel limits to documentation

Ray Chiang created KAFKA-7244: - Summary: Add note about memory map kernel limits to documentation Key: KAFKA-7244 URL: https://issues.apache.org/jira/browse/KAFKA-7244 Project: Kafka Issue Type:

[DISCUSS] KIP-349 Priorities for Source Topics

Hi All, This message is to initiate discussion on a feature to add optional priorities to source topics. Please review https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics Cheers, -- Nick

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, It seems like there's already some confusion about the roles of registry id vs member id. My understanding is that the protocol is currently that the broker assigns a unique "member id" to each consumer in a group, and those consumers must include this id in each subsequent request. It

Build failed in Jenkins: kafka-trunk-jdk10 #362

See Changes: [lindong28] KAFKA-6648; Fetcher.getTopicMetadata() should return all partitions for -- [...truncated 1.54 MB...] kafka.zookeeper.ZooKeeperClientTest > testS

Re: KIP-352: Distinguish URPs caused by reassignment

RIght, that was my thinking too. Ismael On Fri, Aug 3, 2018 at 12:04 PM Gwen Shapira wrote: > On Fri, Aug 3, 2018 at 11:23 AM, Jason Gustafson > wrote: > > > Hey Ismael, > > > > Yeah, my initial inclination was to redefine URP as well. My only doubt > was > > how it would affect existing tools

[jira] [Reopened] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator

[ https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-4479: The code still passes Time.SYSTEM: {code} /* start group coordinator */ // Hardcode Time.SYSTE

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

The KIP has been updated. If it looks good and there are no further comments I'll start a vote early next week. Thanks. --Vahid From: "Vahid S Hashemian" To: dev@kafka.apache.org Date: 08/03/2018 08:19 AM Subject:Re: [DISCUSS] KIP-289: Improve the default group id behavior

Re: KIP-352: Distinguish URPs caused by reassignment

On Fri, Aug 3, 2018 at 11:23 AM, Jason Gustafson wrote: > Hey Ismael, > > Yeah, my initial inclination was to redefine URP as well. My only doubt was > how it would affect existing tools which might depend on URPs to track the > progress of a reassignment. I decided to be conservative in the end,

Re: KIP-352: Distinguish URPs caused by reassignment

Hey Jason, Thanks for the KIP! I have some questions below. The KIP defines "UnderSynchronized" as partitions which have an in-sync replica set that is smaller than the topic's replication factor. Can you clarify how we decide the topic's replication factor? Currently the possible sources include

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hey Guozhang, do you think my understanding makes sense? I would also like to hear from Jason, Mike, John and everyone's opinion on this new approach. Thanks! From: Boyang Chen Sent: Thursday, August 2, 2018 1:41 PM To: dev@kafka.apache.org Subject: Re: [DISCUS

Build failed in Jenkins: kafka-2.0-jdk8 #101

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace >

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

Thanks Vahid! +1 On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian wrote: > Hi everyone, > > I believe the feedback on this KIP has been addressed so far. So I'd like > to start a vote. > The KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 341%3A+Update+Sticky+Assignor%27s+User+

Re: KIP-352: Distinguish URPs caused by reassignment

Hey Ismael, Yeah, my initial inclination was to redefine URP as well. My only doubt was how it would affect existing tools which might depend on URPs to track the progress of a reassignment. I decided to be conservative in the end, but I'd reconsider if we think it is not a major concern. It is an

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

+1 On 8/3/18, 2:13 PM, "Ted Yu" wrote: +1 On Fri, Aug 3, 2018 at 10:40 AM Jason Gustafson wrote: > +1 Thanks Vahid. > > On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > > Hi everyone, > > > > I

[jira] [Created] (KAFKA-7243) Add unit integration tests to validate metrics in Kafka Streams

Guozhang Wang created KAFKA-7243: Summary: Add unit integration tests to validate metrics in Kafka Streams Key: KAFKA-7243 URL: https://issues.apache.org/jira/browse/KAFKA-7243 Project: Kafka

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

+1 On Fri, Aug 3, 2018 at 10:40 AM Jason Gustafson wrote: > +1 Thanks Vahid. > > On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > > Hi everyone, > > > > I believe the feedback on this KIP has been addressed so far. So I'd like > > to start a vote. >

[jira] [Resolved] (KAFKA-3577) Partial cluster breakdown

[ https://issues.apache.org/jira/browse/KAFKA-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang resolved KAFKA-3577. --- Resolution: Duplicate > Partial cluster breakdown > - > > Key

[jira] [Resolved] (KAFKA-3094) Kafka process 100% CPU when no message in topic

[ https://issues.apache.org/jira/browse/KAFKA-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang resolved KAFKA-3094. --- Resolution: Cannot Reproduce > Kafka process 100% CPU when no message in topic >

Re: [VOTE] KIP-341: Update Sticky Assignor's User Data Protocol

+1 Thanks Vahid. On Thu, Aug 2, 2018 at 1:27 PM, Vahid S Hashemian wrote: > Hi everyone, > > I believe the feedback on this KIP has been addressed so far. So I'd like > to start a vote. > The KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 341%3A+Update+Sticky+Assignor%27s+User+D

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Dong, I don't have a preference for the option itself, but we are using " --command-config" in ConfigCommand, ConsumerGroupCommand, BrokerApiVersionsCommand, DeleteRecordsCommand. So it feels like we should use the same here. I think we are currently only using "--config-file" StreamsResetter.

[jira] [Resolved] (KAFKA-4479) Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordinator

[ https://issues.apache.org/jira/browse/KAFKA-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4479. -- Resolution: Won't Fix > Streams tests should pass without hardcoded Time.SYSTEM in GroupCoordin

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hey Rajini, Manikumar, Currently kafka-streams-application-reset.sh uses "--config-file" and a few other tools uses "--command-config". So the config name will be inconsistent no matter which config name we choose. Not sure we currently distinquish between core commands and non-core commands. I am

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hey All, I think I've addressed all pending review. If there is no additional feedback, I'll plan to start a vote thread next week. Thanks, Jason On Tue, Jul 31, 2018 at 9:46 AM, Dong Lin wrote: > Hey Jason, > > Thanks for your reply. I will comment below. > > Regarding 1, we probably can not

[jira] [Created] (KAFKA-7242) Externalized secrets are revealed in task configuration

Bahdan Siamionau created KAFKA-7242: --- Summary: Externalized secrets are revealed in task configuration Key: KAFKA-7242 URL: https://issues.apache.org/jira/browse/KAFKA-7242 Project: Kafka I

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi Jason, Stanislav, Thanks for your feedback. I'll update the KIP later today based on your suggestion. I'll send a note once the update is complete. Thanks. --Vahid From: Stanislav Kozlovski To: dev@kafka.apache.org Date: 08/03/2018 02:34 AM Subject:Re: [DISCUSS] KIP-289: I

[jira] [Created] (KAFKA-7241) Reassignment partitiona to non existent broker

Igor Martemyanov created KAFKA-7241: --- Summary: Reassignment partitiona to non existent broker Key: KAFKA-7241 URL: https://issues.apache.org/jira/browse/KAFKA-7241 Project: Kafka Issue Type

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Updated the KIP. added a note to KIP-340 discussion thread. On Fri, Aug 3, 2018 at 2:52 PM Rajini Sivaram wrote: > Thanks Manikumar. Can you add a note to the KIP-340 discussion thread? > > On Fri, Aug 3, 2018 at 10:04 AM, Manikumar > wrote: > > > Hi Rajini, > > > > Yes, I too prefer using "-

Re: [DISCUSS] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

Hi Dong, In KIP-332 discussion, It was agreed to use "--command-config" option name for passing config property file. We can also use same name in here, to make it consistent across all tools. Thanks, On Thu, Jul 12, 2018 at 9:20 AM Dong Lin wrote: > wiki page is currently read-only and it i

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

Hi, I also believe the minimal change might be better. In my opinion, the less logic we can stuff in the broker - the better. The use ofempty group.id is not ideal and should be discouraged, but I don't inherently believe strict validation in the broker is useful, since it doesn't break anything i

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

Hi Manikumar, +1 (binding) Thanks for the KIP! On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > +1 > > On Thu, Aug 2, 2018 at 7:33 PM Manikumar > wrote: > > > Hi All, > > > > I would like to start voting on KIP-332 which allows AclCommand to use > > AdminClient API for acl management. > > > >

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Thanks Manikumar. Can you add a note to the KIP-340 discussion thread? On Fri, Aug 3, 2018 at 10:04 AM, Manikumar wrote: > Hi Rajini, > > Yes, I too prefer using "--command-config" . Recently in one of the other > KIPs (KIP-340), it was suggested to use > "--config-file", So I just followed th

Re: [VOTE] KIP-342 - Add support for custom SASL extensions in OAuthBearer authentication

Thank you, Jun. Noted in the KIP. On Fri, Aug 3, 2018 at 2:28 AM Jun Rao wrote: > Hi, Stanislav, > > Thanks for the KIP. +1 > > Just one minor comment. Since the JWT token supports customizable claim > fields, it would be useful to clarify when to use the SASL extension vs the > customized field

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Rajini, Yes, I too prefer using "--command-config" . Recently in one of the other KIPs (KIP-340), it was suggested to use "--config-file", So I just followed the recent suggestion. But I think we should use same name in all tools (at least in core tools). If there are no concerns, I will cha

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

Hi Manikumar, We have some tools already (ConfigCommand, ConsumerGroupCommand, DelegationTokenCommand) which use "--command-config" option to specify config file. Perhaps use should use the same name for AclCommand as well? On Thu, Aug 2, 2018 at 7:23 PM, Colin McCabe wrote: > +1 for starting t

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

Thanks John! +1 On Mon, 30 Jul 2018 at 23:58 Guozhang Wang wrote: > Yes, the addendum lgtm as well. Thanks! > > On Mon, Jul 30, 2018 at 3:34 PM, John Roesler wrote: > > > Another thing that came up after I started working on an implementation > is > > that in addition to deprecating "retention"