Re: Case of joining multiple streams/tables

Hi https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup might be worth a look. best regards Patrik On Fri, 6 Dec 2019 at 06:44, Sachin Mittal wrote: > I was thinking more of a builder api at DSL level. > Something like this: > StreamsBuilder.joineBuilder() >

Powered By: add BookingSync and Smily

Hi, Could you please add BookingSync (https://www.bookingsync.com ) and Smily (https://www.smily.com ) to Powered By section?. The description for both: BookingSync: Apache Kafka is used as a backbone of data synchronization and propagating

Re: Powered By: add BookingSync and Smily

Kafka has so many big customers, unless your sites are also the big players, I don’t think they will be added into Kafka official references. Thanks. On Fri, Dec 6, 2019 at 4:50 PM Karol Galanciak wrote: > Hi, > > Could you please add BookingSync (https://www.bookingsync.com < > https://www.boo

Re: Broker Interceptors

Hi, Couldn't this be done without exposing broker internals at the slightly higher level of AbstractRequest and AbstractResponse? Those classes are public. If the observer interface used Java default methods then adding a new request type would not break existing implementations. I'm thinking some

Re: broken cluster / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath

Hello Ismael, my installations of Zookeeper and Kafka are independent, before and after the upgrade. My previous situation was : Zookeeper 3.4.13 and Kafka 2.12-2.1.0 I wanted to update to the latest versions (sounds not unusual) and had this plan : - Zookeeper 3.4.13 --> 3.5.6 - Kafka 2.12-2.1

Kafka consumer group keeps moving to PreparingRebalance and stops consuming

We have a Kafka Streams consumer group that keep moving to PreparingRebalance state and stop consuming. The pattern is as follows: 1. Consumer group is running and stable for around 20 minutes 2. New consumers (members) start to appear in the group state without any clear reason,

Re: Kafka consumer group keeps moving to PreparingRebalance and stops consuming

Hey Avshalom, the consumer instance is initiated per stream thread. You will not be creating new consumers so the root cause is definitely member timeout. Have you changed the max.poll.interval by any chance? That config controls how long you tolerate the interval between poll calls to make sure p

Re: Broker Interceptors

AbstractRequest and AbstractResponse are currently internal classes. Ismael On Fri, Dec 6, 2019 at 1:56 AM Tom Bentley wrote: > Hi, > > Couldn't this be done without exposing broker internals at the slightly > higher level of AbstractRequest and AbstractResponse? Those classes are > public. If

Re: Broker Interceptors

Hi Ismael, How come? They're public in the clients jar. I'm not doubting you, all I'm really asking is "how should I have known this?" Tom On Fri, Dec 6, 2019 at 4:12 PM Ismael Juma wrote: > AbstractRequest and AbstractResponse are currently internal classes. > > Ismael > > On Fri, Dec 6, 2019

Re: Broker Interceptors

Public API classes can be found here: https://kafka.apache.org/24/javadoc/overview-summary.html Everything else is internal. Ismael On Fri, Dec 6, 2019 at 8:20 AM Tom Bentley wrote: > Hi Ismael, > > How come? They're public in the clients jar. I'm not doubting you, all I'm > really asking is

How to set concrete names for state stores and internal topics backed by these

Hi, In my application I have names of internal topics like this: ss-session-application-KSTREAM-JOINOTHER-59-store-changelog-0 ss-session-application-KSTREAM-JOINTHIS-49-store-changelog-0 ss-session-application-KSTREAM-OUTEROTHER-50-store-changelog-0 ss-session-application-

Re: How to set concrete names for state stores and internal topics backed by these

Hi Sachin, The way that Java infers generic arguments makes that case particularly obnoxious. By the way, the problem you're facing is specifically addressed by these relatively new features: * https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+

Re: Powered By: add BookingSync and Smily

Hi Karol, Thanks for submitting these - I'll arrange for a PR to be created to add them to the page. -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 6 Dec 2019 at 08:50, Karol Galanciak wrote: > Hi, > > Could you please add BookingSync (https://www.bookin

Kafka Streams Topology describe format

Hi, I am just posting a section of my topology to basically understand what describe method actually displays. What can we understand just by looking at the topology (like what does --> and <-- arrows represent).

Re: Kafka Streams Topology describe format

Hi again, Sachin, I highly recommend this tool for helping to understand the topology description: https://zz85.github.io/kafka-streams-viz/ I think your interpretation of the format is pretty much spot-on. Hope this helps, -John On Fri, Dec 6, 2019, at 12:21, Sachin Mittal wrote: > Hi, > I am

Re: Broker Interceptors

to me this is an "API vs SPI" discussion. pluggable broker bits should fall on the "SPI" side, where tighter coupling is the price you pay for power and performance, the target audience is small (and supposedly smarter), and compatibility breaks are more common and accepted. On Fri, Dec 6, 2019 at

Static Membership AND Invalid IP addresses forConsumers

I forgot to note that we are using spot instances AND static membership. I was able to confirm that setting application.server with the correct ip-address seems to result in the correct Host being set...however, upon spot-reap/replace the new ip-address does not seem to overwrite the previous i

Re: How to set concrete names for state stores and internal topics backed by these

Hi Sachin We are using a small helper method to keep this readable: private Materialized materializedWith(String name, Serde keySerde, Serde valueSerde) { Materialized materialized = Materialized.as(name); return materialized.withKeySerde(keySerde).withValueSerde(valueSerde); } So the M

Re: [kafka-clients] [VOTE] 2.4.0 RC3

Ran the tests in the following environments using Eric's script available here: https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh OS: Ubuntu 18.04.3 LTS Java Version: OpenJDK 11.0.4 Scala Versions: 2.12.10, 12.13.0, 12.13.1 Gradle Version: 5.6.2 I have made one observation in

Re: [kafka-clients] [VOTE] 2.4.0 RC3

This particular test keeps failing for me ./gradlew :core:integrationTest --tests kafka.admin.ReassignPartitionsClusterTest.shouldMoveSinglePartitionWithinBroker I have created an issue to document my observations https://issues.apache.org/jira/browse/KAFKA-9283 It fails sometimes when the envi

Re: Case of joining multiple streams/tables

Hi Sachin, As Patrik mentioned, KIP-150 is being actively worked on and is likely to be included in the next release. Guozhang On Fri, Dec 6, 2019 at 12:09 AM Patrik Kleindl wrote: > Hi > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup > might > be worth a

What timestamp is used by streams when doing windowed joins

Hi, I have noticed some issues when doing stream to stream windowed joins. Looks like my joined stream does not include all the records. Say I am doing join like this: stream1.join( stream2, (lv, rv) -> ..., JoinWindows.of(Duration.ofMinutes(5)),

Re: What timestamp is used by streams when doing windowed joins

Hi Sachin, I'd need more information to speculate about why your records are missing, but it sounds like you're suspecting something to do with the records' timestamps, so I'll just focus on answering your questions. Streams always uses the same timestamp for all operations, which is the times

How are custom keys compared during joins

Hi, I am facing some weird problems when joining two streams or a stream/table. The joined stream does not contain all the joined records. Also note that my keys are custom keys for which I have implemented equals and hashcode method Is there something else also I need to do to ensure key1 === key

Re: What timestamp is used by streams when doing windowed joins

Hi John, If I check https://docs.confluent.io/current/streams/concepts.html#time It has three notions of time => *event-time*, *processing-time*, *ingestion-time* . If I check https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-timestamp-extractor