[jira] [Created] (KAFKA-8577) Flaky Test `DistributedHerderTest.testJoinLeaderCatchUpFails`

Jason Gustafson created KAFKA-8577: -- Summary: Flaky Test `DistributedHerderTest.testJoinLeaderCatchUpFails` Key: KAFKA-8577 URL: https://issues.apache.org/jira/browse/KAFKA-8577 Project: Kafka

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

Hi Guozhang, 1) This TestRecord is new class in my proposal. So it is a simplified version of ProducerRecord and ConsumerRecord containing only the fields needed to test record content. 2) public final TestInputTopic createInputTopic(final String topicName, final Serde keySerde, final Serde val

Re: [VOTE] KIP-474: To deprecate WindowStore#put(key, value)

Hello everyone, Thanks to who have voted! The final count is :- binding +1: 3 (Matthias, Guozhang, Bill) non-binding +1: 3 (Boyang, Dongjin, John) I'm closing this vote thread. Thanks & Regards Omkar Mestry On Fri, Jun 21, 2019 at 4:36 AM Bill Bejeck wrote: > Sorry for being late to the pa

Build failed in Jenkins: kafka-trunk-jdk8 #3739

See Changes: [wangguoz] KAFKA-8569: integrate warning message under static membership (#6972) -- [...truncated 2.52 MB...] org.apache.kafka.streams.test.OutputVerifierTes

[jira] [Created] (KAFKA-8576) Consumer failed to join the coordinator

yanrui created KAFKA-8576: - Summary: Consumer failed to join the coordinator Key: KAFKA-8576 URL: https://issues.apache.org/jira/browse/KAFKA-8576 Project: Kafka Issue Type: Bug Components:

Re: Partition Reassignment in Cloud

The idea is to expand the cluster, i.e increase the number of brokers and repartition the partition to distribute the load. So at the end of process both the broker needs to be up and running. Would like to know if it is possible to move disk/log.dir from one broker to another and updating metad

Build failed in Jenkins: kafka-trunk-jdk11 #647

See Changes: [wangguoz] KAFKA-8569: integrate warning message under static membership (#6972) -- [...truncated 2.52 MB...] org.apache.kafka.streams.test.OutputVerifierTes

Jenkins build is back to normal : kafka-trunk-jdk8 #3738

See

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

Hi Jun, Thanks for your comments. 1. Yeah I think APIException would not make a distinct call here anymore, and what really matters is the RetriableException. Updated the wiki. 2. Makes sense. Updated the wiki. 3. My current thoughts is to return the first ever hit error for that partition, and

Re: [VOTE] KIP-474: To deprecate WindowStore#put(key, value)

Sorry for being late to the party. I've reviewed the KIP and it's a great step in the right direction. +1 (binding) Thanks, Bill On Thu, Jun 20, 2019 at 11:42 AM John Roesler wrote: > Not that it changes the outcome, but I'm also +1 (nonbinding). > > Been wanting to do this for a while, thank

[jira] [Created] (KAFKA-8575) Investigate cleaning up task suspension

Sophie Blee-Goldman created KAFKA-8575: -- Summary: Investigate cleaning up task suspension Key: KAFKA-8575 URL: https://issues.apache.org/jira/browse/KAFKA-8575 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

Thanks for the feedback, Guozhang and Matthias, Regarding motivation: I'll update the wiki. Briefly: * Any processor can benefit. Imagine a pure user of the ProcessorAPI who has very complex processing logic. I have seen several processor implementation that are hundreds of lines long and call `co

Build failed in Jenkins: kafka-trunk-jdk11 #646

See Changes: [jason] MINOR: Fix Partition::toString method (#6971) -- [...truncated 2.43 MB...] org.apache.kafka.streams.scala.kstream.SuppressedTest > Suppressed.untilWi

Re: [DISCUSS] KIP-439: Deprecate Interface WindowStoreIterator

I have performance concerns about this proposal, because each time a window/session store is accessed, a new (unnecessary) object needs to be created, and accessing the store in the processors is on the hot code path. -Matthias On 6/20/19 10:57 AM, John Roesler wrote: > Hi again, all, > > Afte

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

For encoding the list-type: I see John's point about re-encoding the list-type redundantly. However, I also don't like the idea that the Deserializer returns a fixed type... Maybe it's best allow users to specify the target list type on deserialization via config? Similar for the primitive types:

[jira] [Created] (KAFKA-8574) EOS race condition during task transition leads to LocalStateStore truncation in Kafka Streams 2.0.1

William Greer created KAFKA-8574: Summary: EOS race condition during task transition leads to LocalStateStore truncation in Kafka Streams 2.0.1 Key: KAFKA-8574 URL: https://issues.apache.org/jira/browse/KAFKA-8574

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

Just want to second what Sophie said about the stores. The type of a used stores is completely independent of input/output types. This related to change `addGlobalStore()` method. Why do you want to pin the types? In fact, people request the ability to filter() and maybe even map() the data before

Re: [VOTE] KIP-471: Expose RocksDB Metrics in Kafka Streams

+1 (binding) On 6/20/19 11:53 AM, Guozhang Wang wrote: > +1 (binding) > > Thanks Bruno! > > Would also be interested to see how much overhead it may incur by enabling > DEBUG metrics now, if it is huge we may consider doing finer-grained > metrics enabling, but that would be another follow-up t

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

Hello Jukka, Thanks for writing the KIP, I have a couple of quick questions: 1) Is "TestRecord" an existing class that you propose to piggy-back on? Right now we have a scala TestRecord case class but I doubt that was your proposal, or are you proposing to add a new Java class? 2) Would the new

Re: Partition Reassignment in Cloud

The new broker host meta.properties file can have the broker.id set to the original broker_id (with original host shutdown/decommission), the new host has the storage of the original host (either by copying or by change the network storage mount from original to new host).  This way, it saves t

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

Hi John, Thanks for KIP! I've a few comments below: 1. So far the "Motivation" section is very general, and the only concrete example that I have in mind is `TransformValues#punctuate`. Do we have any other concrete issues that drive this KIP? If not then I feel better to narrow the scope of this

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hey Daniyar, Thanks for looking at it! Something like your screenshot is more along the lines of what I was thinking. Sorry, but I didn't follow what you mean, how would that not be "vanilla java"? Unfortunately the deserializer needs more information, though. For example, what if the inner type

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hey John, I gave read about TypeReference. It could work for the list serde. However, it is not directly supported: https://github.com/FasterXML/jackson-databind/issues/1490 The only way is to pass an actual class object into the const

Re: [VOTE] KIP-471: Expose RocksDB Metrics in Kafka Streams

+1 (binding) Thanks Bruno! Would also be interested to see how much overhead it may incur by enabling DEBUG metrics now, if it is huge we may consider doing finer-grained metrics enabling, but that would be another follow-up task. Guozhang On Wed, Jun 19, 2019 at 1:37 PM Patrik Kleindl wrote:

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi John, Thank you for your input! Yes, my idea looks a little bit over engineered :) I also wanted to see a feedback from Mathias as well since he gave me an idea about storing fixed/variable size entries. Best, Daniyar Yeralin > On Jun 18, 2019, at 6:06 PM, John Roesler wrote: > > Hi Daniy

Re: [DISCUSS] KIP-479: Add Materialized to Join

Hello Bill, Thanks for the KIP. Glad to see that we can likely shooting two birds with one stone. I have some concerns though about those "two birds" themselves: 1. About not breaking compatibility of stream-stream join materialized stores: I think this is a valid issue to tackle, but after think

Re: [DISCUSS] KIP-439: Deprecate Interface WindowStoreIterator

Hi again, all, After wrestling with some other issues around the window interface, I'd propose that we consider normalizing the WindowStore (and SessionStore) interfaces with respect to KeyValueStore. We can't actually make the classes related because they'll clash on the deprecated methods, but w

[jira] [Resolved] (KAFKA-8452) Possible Suppress buffer optimization: de-duplicate prior value

[ https://issues.apache.org/jira/browse/KAFKA-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-8452. - Resolution: Fixed > Possible Suppress buffer optimization: de-duplicate prior value > ---

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

On Thu, Jun 20, 2019, at 00:23, Matthias J. Sax wrote: > Great blog post, Colin! > > Two comments: > > > (1) KIP-258: "future features" -> "the ability to return the latest > timestamp in Interactive Queries" > > This is not a future feature, but the timestamp can be queried in 2.3 > already.

Re: [VOTE] KIP-474: To deprecate WindowStore#put(key, value)

Not that it changes the outcome, but I'm also +1 (nonbinding). Been wanting to do this for a while, thanks Omkar! Now, if we can just get one more binding vote... -John On Sun, Jun 9, 2019 at 1:04 AM omkar mestry wrote: > > Hi, > > Ok the voting thread is still open. > > Thanks Regards > Omkar

[jira] [Created] (KAFKA-8573) kafka-topics.cmd OOM when connecting to a secure cluster without SSL properties

Jorg Heymans created KAFKA-8573: --- Summary: kafka-topics.cmd OOM when connecting to a secure cluster without SSL properties Key: KAFKA-8573 URL: https://issues.apache.org/jira/browse/KAFKA-8573 Project:

[jira] [Created] (KAFKA-8572) Broker reports not leader partition as an error

Antony Stubbs created KAFKA-8572: Summary: Broker reports not leader partition as an error Key: KAFKA-8572 URL: https://issues.apache.org/jira/browse/KAFKA-8572 Project: Kafka Issue Type: Imp

[jira] [Resolved] (KAFKA-8564) NullPointerException when loading logs at startup

[ https://issues.apache.org/jira/browse/KAFKA-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edoardo Comar resolved KAFKA-8564. -- Resolution: Fixed Fix Version/s: 2.2.2 2.1.2 2.3.0

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

Great blog post, Colin! Two comments: (1) KIP-258: "future features" -> "the ability to return the latest timestamp in Interactive Queries" This is not a future feature, but the timestamp can be queried in 2.3 already. (2) Why only listing KIP-428; KIP-445 is equally important. -Matthias