[jira] [Created] (KAFKA-9076) MirrorMaker 2.0 automated consumer offset sync

Ning Zhang created KAFKA-9076: - Summary: MirrorMaker 2.0 automated consumer offset sync Key: KAFKA-9076 URL: https://issues.apache.org/jira/browse/KAFKA-9076 Project: Kafka Issue Type: Improvemen

Build failed in Jenkins: kafka-trunk-jdk8 #3988

See Changes: [github] KAFKA-8940: Tighten up SmokeTestDriver (#7565) [matthias] MINOR: don't require key serde in join materialized (#7557) -- [...truncated 7.72 MB...]

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

+1 (non-binding). Thanks! On Tue, Oct 22, 2019 at 1:03 AM Gwen Shapira wrote: > +1 > > Thanks for leading this :) > > On Mon, Oct 21, 2019 at 3:43 PM Colin McCabe wrote: > > > > Hi all, > > > > I'd like to start the vote on KIP-538: Add a metric tracking the number > of open connections with a

Build failed in Jenkins: kafka-2.4-jdk8 #32

See Changes: [matthias] MINOR:Upgrade guide updates for KIP-479 (#7550) -- [...truncated 4.95 MB...] kafka.admin.PreferredReplicaLeaderElectionCommandTest > testPreferredRep

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

Solid KIP. This also has the side-effect of bumping zookeeper.connection.timeout.ms to a more reasonable default (the 18s session timeout). Perhaps worth mentioning +1 (non-binding) On Tue, Oct 22, 2019 at 5:47 AM Ismael Juma wrote: > +1 (binding) > > On Mon, Oct 21, 2019, 5:28 PM Jason Gustafs

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

Thanks for the KIP, Colin! This is a good idea, I think it makes total sense. We also have the max-size tunables on the producer-side (max.request.size). Do we have a broker equivalent denoting the maximum size of a produce request? Is socket.request.max.bytes expected to act as that and if not, i

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

+1 binding. This kip will possibly fix a lot of extra line of configuration. Thanks, On Tue, 22 Oct 2019 at 09:06, Stanislav Kozlovski wrote: > Solid KIP. This also has the side-effect of bumping > zookeeper.connection.timeout.ms to a more reasonable default (the 18s > session timeout). Perhaps

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

Hi Mickael, Thanks for the KIP. The use of String to represent the desired state in the API seems less typesafe than would be ideal. Is there a reason not to use the existing ConsumerGroupState enum (even if the state is serialized as a String)? While you say that the list-of-names result from l

Re: [DISCUSS] KIP-280: Enhanced log compaction

Hi Senthilnathan, In the motivation isn't it a little misleading to say "On the producer side, we clearly preserve an order for the two messages, "? IMHO, the semantics of the producer are clear that having an observed order of sending records from different producers is not sufficient to guarant

[jira] [Created] (KAFKA-9077) System Test Failure: StreamsSimpleBenchmarkTest

Manikumar created KAFKA-9077: Summary: System Test Failure: StreamsSimpleBenchmarkTest Key: KAFKA-9077 URL: https://issues.apache.org/jira/browse/KAFKA-9077 Project: Kafka Issue Type: Bug Aff

[jira] [Created] (KAFKA-9078) System Test Failure: ConnectRestApiTest

Manikumar created KAFKA-9078: Summary: System Test Failure: ConnectRestApiTest Key: KAFKA-9078 URL: https://issues.apache.org/jira/browse/KAFKA-9078 Project: Kafka Issue Type: Bug Affects Ver

[jira] [Created] (KAFKA-9079) System Test Failure: TransactionsTest

Manikumar created KAFKA-9079: Summary: System Test Failure: TransactionsTest Key: KAFKA-9079 URL: https://issues.apache.org/jira/browse/KAFKA-9079 Project: Kafka Issue Type: Bug Affects Versi

[jira] [Created] (KAFKA-9080) System Test Failure: MessageFormatChangeTest.testCompatibilty

Manikumar created KAFKA-9080: Summary: System Test Failure: MessageFormatChangeTest.testCompatibilty Key: KAFKA-9080 URL: https://issues.apache.org/jira/browse/KAFKA-9080 Project: Kafka Issue Ty

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

Thanks Jason for the KIP. +1 (non-binding). I assume this change will be added to the upgrade notes as the new values are effective for clusters that were using earlier defaults. Cluster may get into the race condition of having lower replica.lag.time.max.ms on a leader than zookeeper.session.tim

Build failed in Jenkins: kafka-trunk-jdk8 #3989

See Changes: [matthias] MINOR: KIP-307 upgrade guide docs (#7547) [matthias] MINOR:Upgrade guide updates for KIP-479 (#7550) -- [...truncated 5.47 MB...] org.apache.kafk

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

Hi Colin, Thanks for the KIP. This is definitely needed. Have we considered having a lower max limit by default? 75 MB is pretty high. Did we consider something like 55 MB, which is 10% above the default for consumers? Ismael On Mon, Oct 21, 2019 at 2:57 PM Colin McCabe wrote: > Hi all, > > I

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

Hi Richard, Thanks for the KIP. After reading it over, I'm wondering if you saw this one: https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time It seems like it overlaps with your use case, where you're willing to get event

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

One more thought: 14) Is specifying the allowed lag in number of records a useful way for users to declare how stale an instance is allowed to be? Would it be more intuitive for users to specify the allowed lag in time units (would event time or processing time be better)? It seems hard for users

Build failed in Jenkins: kafka-1.1-jdk7 #275

See Changes: [rhauch] MINOR: Fix Java 8 dependency from KAFKA-6290 backport (#7572) -- Started by an SCM change Running as SYSTEM [EnvInject] - Loading node environment vari

Re: [DISCUSS] KIP-506: Allow setting SCRAM password via Admin interface

Hi Viktor and Colin, Thanks for taking a look. The intent behind sasl.scram.password.change.enabled was to give the admins the choice about whether they're willing for passwords to be sent to the broker in the clear (after all they know their deployment and network best). I'm fine with removing i

[jira] [Created] (KAFKA-9081) Loosen reassignment completion to handle existing URPs

Jason Gustafson created KAFKA-9081: -- Summary: Loosen reassignment completion to handle existing URPs Key: KAFKA-9081 URL: https://issues.apache.org/jira/browse/KAFKA-9081 Project: Kafka Issu

Build failed in Jenkins: kafka-1.1-jdk7 #276

See Changes: [rhauch] KAFKA-9057: Backport KAFKA-8819 and KAFKA-8340 before 2.0 (#7549) -- Started by an SCM change Running as SYSTEM [EnvInject] - Loading node environment

Build failed in Jenkins: kafka-trunk-jdk8 #3990

See Changes: [manikumar] MINOR: Add upgrade docs for 2.4.0 release -- [...truncated 2.70 MB...] org.apache.kafka.streams.test.TestRecordTest > testFields PASSED org.apac

Re: [VOTE] 2.3.1 RC2

+1 I ran the basic quickstart on the 2.12 artifact and verified signatures/checksums. I also looked over the release notes. I see that KAFKA-8950 is included, so maybe they just need to be refreshed. Thanks for running the release! -Jason On Fri, Oct 18, 2019 at 5:23 AM David Arthur wrote: >

[DISCUSS] KIP-543: Expand ConfigCommand's non-ZK functionality

Hello all, I wrote a KIP about expanding the ConfigCommand's functionality when not accessing ZooKeeper directly, i.e. when --bootstrap-servers is specified instead of --zookeeper. This should bring the two into parity. There's also a bit of bikeshedding about additional flags for shortening enti

[jira] [Resolved] (KAFKA-4113) Allow KTable bootstrap

[ https://issues.apache.org/jira/browse/KAFKA-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-4113. Resolution: Invalid Closing this ticket as "invalid" because KTables are populated based on

Re: [VOTE] 2.3.1 RC2

Thanks, Jonathon and Jason. I've updated the release notes along with the signature and checksums. KAFKA-9053 was also missing. On Tue, Oct 22, 2019 at 3:47 PM Jason Gustafson wrote: > +1 > > I ran the basic quickstart on the 2.12 artifact and verified > signatures/checksums. > > I also looked o

[jira] [Created] (KAFKA-9082) Move ConfigCommand to use KafkaAdminClient APIs

Brian Byrne created KAFKA-9082: -- Summary: Move ConfigCommand to use KafkaAdminClient APIs Key: KAFKA-9082 URL: https://issues.apache.org/jira/browse/KAFKA-9082 Project: Kafka Issue Type: Improve

[jira] [Created] (KAFKA-9083) Various parsing issues in Values class

Chris Egerton created KAFKA-9083: Summary: Various parsing issues in Values class Key: KAFKA-9083 URL: https://issues.apache.org/jira/browse/KAFKA-9083 Project: Kafka Issue Type: Bug

Build failed in Jenkins: kafka-2.4-jdk8 #33

See Changes: [manikumar] MINOR: Add upgrade docs for 2.4.0 release -- [...truncated 5.23 MB...] kafka.tools.DumpLogSegmentsTest > testPrintDataLog STARTED kafka.tools.DumpL

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

+1 (binding). Thanks, Jason. best, Colin On Mon, Oct 21, 2019, at 17:27, Jason Gustafson wrote: > I'd like to start a vote for KIP-537: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout > . > > +1 from me > > Thanks, > Jason >

[jira] [Created] (KAFKA-9084) AlterPartitionReassignments should not allow all replicas to be removed

Colin McCabe created KAFKA-9084: --- Summary: AlterPartitionReassignments should not allow all replicas to be removed Key: KAFKA-9084 URL: https://issues.apache.org/jira/browse/KAFKA-9084 Project: Kafka

[jira] [Resolved] (KAFKA-9084) AlterPartitionReassignments should not allow all replicas to be removed

[ https://issues.apache.org/jira/browse/KAFKA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9084. - Resolution: Fixed Resolved by https://github.com/apache/kafka/pull/7574 > AlterPartitionReassign

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

+1 Thanks Colin. On Tue, Oct 22, 2019 at 12:42 AM Tom Bentley wrote: > +1 (non-binding). Thanks! > > On Tue, Oct 22, 2019 at 1:03 AM Gwen Shapira wrote: > > > +1 > > > > Thanks for leading this :) > > > > On Mon, Oct 21, 2019 at 3:43 PM Colin McCabe wrote: > > > > > > Hi all, > > > > > > I'd l

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi, Harsha, Historically, we tried using a feature branch in 0.8. The experience actually wasn't great. Merging the feature branch to the main branch required additional review work and each merge with the main branch added the risk of introducing new bugs. So, we have been avoiding feature branch

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

+1 (non-bonding) it’s a helpful info. Thanks for the KIP. Regards, On Tue, 22 Oct 2019 at 23:45, Jason Gustafson wrote: > +1 Thanks Colin. > > On Tue, Oct 22, 2019 at 12:42 AM Tom Bentley wrote: > > > +1 (non-binding). Thanks! > > > > On Tue, Oct 22, 2019 at 1:03 AM Gwen Shapira wrote: > > >

Re: [DISCUSS] KIP-543: Expand ConfigCommand's non-ZK functionality

Hi Brian, Thanks for the KIP! This looks like a nice cleanup. It will be good to be able to perform these operations without direct ZK access. I guess this is a nitpick, but for the operations proposed for later, it might be nice to have a different color, like yellow, to indicate that we'll

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

+1 (binding). Thanks, Harsha On Tue, Oct 22, 2019 at 3:20 PM Colin McCabe wrote: > +1 (binding). > > Thanks, Jason. > > best, > Colin > > > On Mon, Oct 21, 2019, at 17:27, Jason Gustafson wrote: > > I'd like to start a vote for KIP-537: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

Hi John, Reading the KIP over, I actually think that this will be an excellent extension to what we have now. I've read the discussion for that KIP and it looks like Jonathan never mentioned how time would be tracked (wall-clock based that is) while this KIP I believe proposes a feasible approach

Re: [DISCUSS] KIP-543: Expand ConfigCommand's non-ZK functionality

Sure - updated. Thanks, Brian On Tue, Oct 22, 2019 at 4:02 PM Colin McCabe wrote: > Hi Brian, > > Thanks for the KIP! This looks like a nice cleanup. It will be good to > be able to perform these operations without direct ZK access. > > I guess this is a nitpick, but for the operations propos

Jenkins build is back to normal : kafka-trunk-jdk11 #903

See

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Jun, Thanks for the feedback. Given the no.of engineers involved in cross-team effort it would be great to have this as feature branch. Irrespective of if its in my fork or in Apache Kafka's branch it needs to be constantly rebased from trunk to keep it current. Our proposal is to mer

[jira] [Created] (KAFKA-9085) Admin client should propagate UnsupportedVersion exception to user instead of crashing

Boyang Chen created KAFKA-9085: -- Summary: Admin client should propagate UnsupportedVersion exception to user instead of crashing Key: KAFKA-9085 URL: https://issues.apache.org/jira/browse/KAFKA-9085 Proj

Build failed in Jenkins: kafka-trunk-jdk11 #904

See Changes: [github] MINOR: Reset timer when all the buffer is drained and empty (#7573) [cmccabe] MINOR: Check against empty replicas in AlterPartitionReassignments [cmccabe] MINOR: Rename brokers to replicas

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Hey Walker, Thanks for the KIP! I have just a couple of questions: 1) It seems a little awkward to me that with the current API, we have a nearly identical "add stream to cogroup" method, except for the first which has a different signature (ie the first stream is joined as stream.cogroup(Aggrega

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

>Regarding the HDFS dependency its not a direct dependency rather its implementing the RemoteStorageManager interface. We packaged it along with core to make it more convenient to test it. We can move this to external module and keep it there. Let me know what you think. Let me elaborate more on t

Build failed in Jenkins: kafka-2.4-jdk8 #34

See Changes: [cmccabe] MINOR: Check against empty replicas in AlterPartitionReassignments [cmccabe] MINOR: Rename brokers to replicas in the reassignment API (#7570) -- [...

Re: [Discuss] - KIP-532 - Add KStream#toTable to the Streams DSL

Any update on this KIP? On 10/7/19 3:06 PM, Matthias J. Sax wrote: > Aishwarya, > > Why is bullet point (2) formatted as "strike through"? If you intend to > replace it with bullet point (3), just remove it completely. The KIP > should reflect the actual proposal. Maybe move it to "Rejected > Alt

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

Any update on this KIP? On 10/7/19 3:35 PM, Matthias J. Sax wrote: > Sorry for the late reply. The 2.4 deadline kept us quite busy. > > About `StateStoreMigratedException`: > > Why is it only thrown if the state is REBALANCING? A store might be > migrated during a rebalance, and Kafka Streams mi

Build failed in Jenkins: kafka-trunk-jdk8 #3991

See Changes: [github] MINOR: Reset timer when all the buffer is drained and empty (#7573) -- [...truncated 7.41 MB...] org.apache.kafka.streams.kstream.internals.graph.S

Re: [DISCUSS] KIP-530: Considerrenaming'UsePreviousTimeOnInvalidTimeStamp' classto'UsePartitionTimeOnInvalidTimeStamp'

Thanks for the KIP, Rabi. Overall LGTM, but I think we can simplify the write up a little bit. Note that `ExtractRecordMetadataTimestamp` is not a public class (and I am actually wondering why `ExtractRecordMetadataTimestamp#onInvalidTimestamp` is actually public? Seems it could be protected or e

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

bump! Clement/Rajini? Any responses based on the latest posts? On Wed, Oct 16, 2019 at 10:58 PM Maulin Vasavada wrote: > bump! > > On Sun, Oct 13, 2019 at 11:16 PM Maulin Vasavada < > maulin.vasav...@gmail.com> wrote: > >> Hi Clement >> >> 1) existing validation code will remain in SslFactory >>

Build failed in Jenkins: kafka-trunk-jdk11 #905

See Changes: [github] MINOR: Add toString to PartitionReassignment (#7579) -- [...truncated 2.71 MB...] org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED