Re: Subject: [VOTE] 2.4.1 RC0

Hi Bill, I built from source and ran unit and integration tests. They passed. There was a large number of skipped tests, but I'm assuming that is intentional. Cheers Eno On Tue, Mar 3, 2020 at 8:42 PM Eric Lalonde wrote: > > Hi, > > I ran: > $

Operationalizing Zookeeper and common gotchas

Hi folks, The team here has come up with a couple of clarifying tips for operationalizing Zookeeper for Kafka that we found missing from the official documentation, and passed them along to share. If you find them useful, I'm thinking of putting on

Re: [VOTE] 2.1.1 RC2

+1 passes unit + integration tests. Eno On Fri, Feb 8, 2019 at 11:10 PM Magnus Edenhill wrote: > +1 > > Passes librdkafka test suite. > > Den fre 8 feb. 2019 kl 21:02 skrev Colin McCabe : > > > Hi all, > > > > This is the third candidate for release of Apache Kafka 2.1.1. This > > release

Re: [VOTE] 2.1.1 RC1

ins build we did here: > https://builds.apache.org/job/kafka-2.1-jdk8/118/ > > Perhaps there is an environment issue at play here? Do you get the same > failures running those tests on the 2.1 release? > > Best, > Colin > > On Wed, Jan 30, 2019, at 09:11, Eno Thereska wrote: &

Re: [VOTE] 2.1.1 RC1

Hi Colin, I've been running the tests and so far I get the following failures. Are they known? kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow FAILED kafka.server.ReplicaManagerQuotasTest > testCompleteInDelayedFetchWithReplicaThrottling FAILED

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

The general answer depends on what control plane software is taking care of your Kafka deployment. You probably have a layer that launches Kafka instances and monitors their health, right? If so, that layer should take care of the mapping between instances and broker IDs and keep that in a table

Re: [VOTE] 2.1.0 RC1

Built code and ran tests. Getting a single integration test failure: kafka.log.LogCleanerParameterizedIntegrationTest > testCleansCombinedCompactAndDeleteTopic[3] FAILED java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 ->

Re: [VOTE] 2.0.1 RC0

Anything else holding this up? Thanks Eno On Thu, Nov 1, 2018 at 10:27 AM Jakub Scholz wrote: > +1 (non-binding) ... I used the staged binaries and run tests with > different clients. > > On Fri, Oct 26, 2018 at 4:29 AM Manikumar > wrote: > > > Hello Kafka users, developers and

Re: Problems trying to make kafka 'rack-aware'

Hi Bryan, I did a simple check with starting a broker with no rack id and then restarting with a rack id and I can confirm I could get the rack id from zookeeper after the restart. This was on trunk. Does that basic check work for you (i.e., without reassigning partitions)? Thanks Eno On Fri,

Re: Continue to consume messages when exception occurs in Kafka Stream

Hi Duy, What kind of exception are you getting? With KIP-161 (checked in trunk) we allow log-and-skip type exception handlers for deserialization errors: https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers

Re: Forwarding consumer with kafka streams

Hi Ricardo, Kafka Streams should handle that case as well. What streams config are you using, could you share it? There is one parameter that is called “ConsumerConfig.AUTO_OFFSET_RESET_CONFIG” and by default it’s set to “earliest”. Any chance your app has changed it to “latest”? Thanks Eno

Re: Kafka Streams not auto-creating the state store changelog topic

t there were no > changes. > Also, if I have regex for the input topic on kafka streams and a new topic > is added to kafka matching the regex, the application will break? > > On Fri, Aug 4, 2017, 8:33 PM Eno Thereska <eno.there...@gmail.com > <mailto:eno.there...@gmail.co

Re: Kafka Streams not auto-creating the state store changelog topic

> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582) > at > org.apache.k

Re: Kafka Streams not auto-creating the state store changelog topic

Hi Anish, Could you give more info on how you create the state stores in your code? Also could you copy-paste the exact error message from the log? Thanks Eno > On Aug 4, 2017, at 9:05 AM, Anish Mashankar wrote: > > I have a new application, call it streamsApp with

Re: Do we have to query localWindowStore in same java instance we are creating the store

Hi Sachin, 1. You can run a remote query and we provide some example code (https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/ ) however by default

Re: Windows OS platform support

Hi Harish, I believe many people/orgs use it on Windows. We rely on the community to test/fix/answer any Windows questions, same as with Linux or MacOS. However, based on what I've observed, perhaps there are more people answering Linux-related questions. Eno > On 14 Jul 2017, at 13:24,

Re: State management & restore functionality

io/current/streams/developer-guide.html#streams-developer-guide-standby-replicas <http://docs.confluent.io/current/streams/developer-guide.html#streams-developer-guide-standby-replicas>. Hope this helps, Eno > regards. > > On Fri, Jul 14, 2017 at 6:23 PM, Eno Thereska <eno.the

Re: Kafka streams: Record linkage / collecting all messages linked to one entity

key is a node. Connect the > messages with the match-keys using edges. Now the message nodes are connected > through the match-key nodes. Each entity is defined by the graph that > connects all messages that are linked together. > > Kind regards, > > Wladislaw >

Re: State management & restore functionality

Hi Debasish, Your intuition about the first part is correct. Kafka Streams automatically assigns a partition of a topic to a task in an instance. It will never be the case that the same partition is assigned to two tasks. About the merging or changing of partitions part, it would help if we

Re: Is this a decent use case for Kafka Streams?

From just looking at your description of the problem, I'd say yes, this looks like a typical scenario for Kafka Streams. Kafka Streams supports exactly once semantics too in 0.11. Cheers Eno > On 12 Jul 2017, at 17:06, Stephen Powis wrote: > > Hey! I was hoping I could

Re: Kafka streams: Record linkage / collecting all messages linked to one entity

Hi Wladislaw, Would splitting the one topic into multiple topics be acceptable at all? E.g., you could use the "branch" function in the DSL to split the messages and send to different topics. Then, once you have multiple topics you can do the joins etc. Thoughts? Thanks Eno > On 11 Jul

Re: kafka-streams app(s) stopped consuming new events

It’s hard to tell, the logs do not contain much, I agree. It could be a number of things. If it’s happening as you say on restart as well (so it’s reproducible), any chance you could start streaming with DEBUG logs on and collect those logs? I’m hoping something shows up there. Thanks, Eno

Re: Kafka Stream invalid partitions

; > > > I use this API as follows: > > ... > .groupByKey() > .aggregate(...) > .toStream(...); > > > Please let me know if you need addiotional information. > > Thanks, > > > 2017-06-27 11:39 GMT+02:00 Eno Thereska <eno.there...@gmail.com>:

Re: Kafka Stream invalid partitions

Hi there, Thanks for the report. What version of Kafka are you using? Also, between runs do you change the number of partitions for your topics? I’m trying to figure out how this problem happens, any information on what is changing in between runs is appreciated. Thanks, Eno > On Jun 27,

Re: Kafka 10.2.1 KStreams rebalancing issue

Hi Sameer, Could you elaborate on your question? Are you concerned that machine2 does not have any tasks in the beginning? Could you share your streams configuration? In particular how many threads does each stream instance have? Also how many topics and partitions do you have? Thanks, Eno

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

ere is no need for the response config option. Thanks Eno > > Guozhang > > > > > > > > > On Wed, Jun 21, 2017 at 1:39 AM, Eno Thereska <eno.there...@gmail.com > <mailto:eno.there...@gmail.com>> > wrote: > >> Thanks

Re: [DISCUSS] Streams DSL/StateStore Refactoring

the proposed > APIs? > > Meanwhile Damian could you provide a formal set of APIs for people to > exercise on them? Also could you briefly describe how custom storage > engines could be swapped in with the above APIs? > > > > Guozhang > > > On Wed, Jun 21, 2017 at

Re: [DISCUSS] Streams DSL/StateStore Refactoring

se is >> specified as you have done above. >> >> On Wed, 21 Jun 2017 at 16:24 Eno Thereska <eno.there...@gmail.com> wrote: >> >>> (cc’ing user-list too) >>> >>> Given that we already have StateStoreSuppliers that are configurable >> usin

Re: [DISCUSS] Streams DSL/StateStore Refactoring

(cc’ing user-list too) Given that we already have StateStoreSuppliers that are configurable using the fluent-like API, probably it’s worth discussing the other examples with joins and serdes first since those have many overloads and are in need of some TLC. So following your example, I guess

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

nd it in the future and probably would like to not rename the > config again. Also from the experience of `default partitioner` and > `default timestamp extractor` we may also make sure that the passed in > object can be either a string "class name" or a class object? > > >

Re: KStream Usage spikes memory consumption and breaks Kafka

Could you provide some configuration information and more context? What application are you running, when is it running out of memory? Otherwise it's hard to tell. Eno > On 20 Jun 2017, at 22:15, IT Consultant <0binarybudd...@gmail.com> wrote: > > Hi All , > > Kafka instance is breaking down

Re: [VOTE] 0.11.0.0 RC1

+1 (non-binding) passes Kafka Streams tests. Thanks, Eno > On 19 Jun 2017, at 06:49, Magnus Edenhill wrote: > > +1 (non-binding) > > Passes librdkafka integration tests (v0.9.5 and master) > > > 2017-06-19 0:32 GMT+02:00 Ismael Juma : > >> Hello Kafka

Re: Scala type mismatch after upgrade to 0.10.2.1

Hi there, Yeah with 0.10.2 some Scala applications need to explicitly declare the type of certain variables. See this: http://docs.confluent.io/current/streams/upgrade-guide.html#scala Thanks Eno > On Jun 17, 2017, at 10:03

Re: Slow Consumer Group Startup

m> wrote: > >> While I do have some logs its not trivial to share since the logs are >> across 16 JVMs and a few different hosts. >> >> On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska <eno.there...@gmail.com> >> wrote: >> >>> The delay in that K

Re: Single Key Aggregation

I'm not sure if I fully understand this but let me check: - if you start 2 instances, one instance will process half of the partitions, the other instance will process the other half - for any given key, like key 100, it will only be processed on one of the instances, not both. Does this help?

Re: KStream and KTable different behaviour on filter() operation

Yeah the semantics are slightly different. For a KTable, a null value just means that the record is a tombstone, and will be anyways ignored by subsequent processing:

Re: Kafka Streams vs Spark Streaming : reduce by window

> > Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> &

Re: Kafka Streams vs Spark Streaming : reduce by window

Hi Paolo, That is indeed correct. We don’t believe in closing windows in Kafka Streams. You could reduce the number of downstream records by using record caches: http://docs.confluent.io/current/streams/developer-guide.html#record-caches-in-the-dsl

Re: Question on Support provision

Hi Sofia, Thank you for your recent enquiry for Kafka support services. Confluent employs some of the world’s foremost Apache Kafka experts, and that expertise shows in the level of support we can provide. The subscription offers a

Re: Slow Consumer Group Startup

The delay in that KIP is just 3 seconds, not minutes though, right? Would you have any logs to share? Thanks Eno > On 14 Jun 2017, at 16:14, Bryan Baugher wrote: > > Our consumer group isn't doing anything stateful and we've seen this > behavior for existing groups as well.

Re: getting intermittent TimeoutException at producer side in streams application

Hi Sachin, As Damian mentioned it'd be useful to see some logs from both broker and streams. One thing that comes to mind is whether your topics are replicated at all. You could try setting the replication factor of streams topics (e.g., changelogs and repartition topics) to 2 or 3 using

Re: Kafka Streams Failed to rebalance error

your running within each instance? >>>>> >>>>> Guozhang >>>>> >>>>> >>>>> On Wed, Jun 7, 2017 at 3:18 PM, João Peixoto <joao.harti...@gmail.com >>>> >>>>> wrote: >>>>> >&

Re: Kafka Streams Failed to rebalance error

Hi there, This might be a bug, would you mind opening a JIRA (copy-pasting below is sufficient). Thanks Eno > On 7 Jun 2017, at 21:38, João Peixoto wrote: > > I'm using Kafka Streams 0.10.2.1 and I still see this error > > 2017-06-07 20:28:37.211 WARN 73 --- [

Re: Reliably implementing global KeyValueStore#get

our metadata lookup > completes* > >> >> HTH, >> Damian >> >> On Tue, 6 Jun 2017 at 18:11 Steven Schlansker <sschlans...@opentable.com> >> wrote: >> >>> >>>> On Jun 6, 2017, at 6:16 AM, Eno Thereska <eno.there...@gmail

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

Comments inline: > On 5 Jun 2017, at 18:19, Jan Filipiak <jan.filip...@trivago.com> wrote: > > Hi > > just my few thoughts > > On 05.06.2017 11:44, Eno Thereska wrote: >> Hi there, >> >> Sorry for the late reply, I was out this pas

Re: Reliably implementing global KeyValueStore#get

Hi Steven, Do you know beforehand if a key exists? If you know that and are getting null() the code will have to retry by refreshing the metadata and going to the new instance. If you don’t know beforehand if a key exists or not you might have to check all instances of a store to make sure.

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

Hi there, Sorry for the late reply, I was out this past week. Looks like good progress was made with the discussions either way. Let me recap a couple of points I saw into one big reply: 1. Jan mentioned CRC errors. I think this is a good point. As these happen in Kafka, before Kafka Streams

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

ded a bad > deserializer to a method that creates a state store. In which case it would > be a user error and probably should just abort? > > Thanks, > Damian > > On Fri, 26 May 2017 at 16:32 Eno Thereska <eno.there...@gmail.com> wrote: > >> See latest reply t

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

to allow for two > handlers instead of one? One for deserialization exception and one for > all other exceptions from user code? > > Just a thought. > > > -Matthias > > On 5/26/17 7:49 AM, Jim Jagielski wrote: >> >>> On May 26, 2017, at 5:1

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

ndlerResponse handle(that guy here >>>>>>> ConsumerRecord<||byte||[], > ||byte||[]> record, Exception exception);| > |}| > || > |public| |enum| |HandlerResponse {| > |||/* continue with processing */| > |||CONTINUE(||1||), | > |||/* fail the pro

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

> processor throwing while processing. > > would it not make more sense to get the ProcessorName + object object for > processing and > statestore or topic name + byte[] byte[] for serializers? maybe passing in > the used serdes? > > Best Jan > > > > On 25.05

Re: Streams error handling

error with the key/value prior > to the error and handle it - possibly by replacing the message, sending a > message to a new stream, or even putting it back on the original stream for > retry. > > Regards > MikeG > > On Wed, 24 May 2017 at 10:09, Eno Thereska <eno.there...@gmail.

[DISCUSS]: KIP-161: streams record processing exception handlers

Hi there, I’ve added a KIP on improving exception handling in streams: KIP-161: streams record processing exception handlers. https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+record+processing+exception+handlers

Re: Streams error handling

or > similar methods would be able to examine the error with the key/value prior > to the error and handle it - possibly by replacing the message, sending a > message to a new stream, or even putting it back on the original stream for > retry. > > Regards > MikeG > >

Re: Streams error handling

Just a heads up that we're tracking this and other improvements in exception handling at https://issues.apache.org/jira/browse/KAFKA-5156 . Thanks Eno > On 23 May 2017, at 17:31, Mike Gould wrote: > > That's great for the

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

; one core, so I don't see how it could be CPU starved. Still I will of > course try your suggestion. > > Thanks again, > V. > > > On Tue, May 16, 2017 at 5:15 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > >> Which version of Kafka are you using? It migh

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

; and new ones appear regularly. > I can certainly try streams 0.10.2.1 if it's compatible with Kafka 0.10.1. > I have not checked the compatibility matrix yet. > > Thanks for the help, > V. > > On Tue, 16 May 2017 at 17:57, Eno Thereska <eno.there...@gmail.com> wrot

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

itions and only uses about 100% of >> one core, so I don't see how it could be CPU starved. Still I will of >> course try your suggestion. >> >> Thanks again, >> V. >> >> >> On Tue, May 16, 2017 at 5:15 PM, Eno Thereska <eno.there...@gmail.com

Re: KafkaStreams reports RUNNING even though all StreamThreads has crashed

Hi Andreas, Thanks for reporting. This sounds like a bug, but could also be a semantic thing. Couple of questions: - which version of Kafka are you using? - what is the nature of the failure of the threads, e.g., how come they have all crashed? If all threads crash, was there an exception they

Re: Can state stores function as a caching layer for persistent storage

tition" topic actually solves this problem. >>>> >>>> There's very little information about the latter, at least that I could >>>> find, but an entry seems to be added whenever a record enters the >>>> "aggregate", but the state stor

Re: Kafka Streams RocksDB permanent StateStoreSupplier seemingly never deletes data

Which version of Kafka are you using? It might be that RocksDb doesn't get enough resources to compact the data fast enough. If that's the case you can try increasing the number of background compaction threads for RocksDb through the RocksDbConfigSetter class (see

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

+1 thanks. Eno > On 16 May 2017, at 04:20, BigData dev wrote: > > Hi All, > Given the simple and non-controversial nature of the KIP, I would like to > start the voting process for KIP-157: Add consumer config options to > streams reset tool > >

Re: Kafka-streams process stopped processing messages

was running without any >>> problem for a few hours. Now, each time I restart this process it gets >> into >>> rebalancing state for several hours. >>> That mean that every time we need to deploy a new version of our app >> (which >>> can be several t

Re: Kafka Streams reports: "The timestamp of the message is out of acceptable range"

Hi Frank, Could you confirm that you're using 0.10.2.1? This error was fixed ad part of this JIRA I believe: https://issues.apache.org/jira/browse/KAFKA-4861 Thanks Eno > On 14 May 2017, at 23:09, Frank Lyaruu wrote: > >

Re: Can state stores function as a caching layer for persistent storage

Hi João, Some answers inline: > On 12 May 2017, at 18:27, João Peixoto wrote: > > Thanks for the comments, here are some clarifications: > > I did look at interactive queries, if I understood them correctly it means > that my state store must hold all the results in

Re: Can state stores function as a caching layer for persistent storage

Hi there, A couple of general comments, plus some answers: - general comment: have you thought of using Interactive Queries to directly query the aggregate data, without needing to store them to an external database (see this blog:

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

+1 for me. I’m not sure we even need a KIP for this but it’s better to be safe I guess. Eno > On May 9, 2017, at 8:41 PM, BigData dev wrote: > > Hi, Everyone, > > Since this is a relatively simple change, I would like to start the voting > process for KIP-156: Add

Re: Kafka Stream stops polling new messages

t we just > increased to guard against failure for long stat recreation phases. > > Any thoughts? > > > -Matthias > > > On 5/3/17 8:48 AM, João Peixoto wrote: > > That'd be great as I'm not familiar with the protocol there > > On Wed, May 3, 2017 at 8:41 AM

Re: Large Kafka Streams deployment takes a long time to bootstrap

Hi there, I wanted to add something: how many CPU cores does each of your Kubernetes instance have? In 0.10.2.1 we noticed a regression in environments with 1 core as described in https://issues.apache.org/jira/browse/KAFKA-5174 . If you have

Re: Kafka-streams process stopped processing messages

iti <shim...@gmail.com> wrote: > > Thanks Eno, > I already set the the recurve buffer size to 1MB > I will also try producer > > What about session timeout and heart beat timeout? Do you think it should > be increased? > > Thanks, > Shimi > > On Sat, 6 May 2017

Re: Kafka-streams process stopped processing messages

sa-events 1 1078428 >>> 1078428 0 >>> sa-4557bf2d-ba79-42a6-aa05-5b4c9013c022-StreamThread-1-consumer_/ >> 10.0.10.9 >>> sa sa-events 2 unknown >>> 26093910unknown >>

Re: Windowed aggregations memory requirements

This is a timely question and we've updated the documentation here on capacity planning and sizing for Kafka Streams jobs: http://docs.confluent.io/current/streams/sizing.html . Any feedback welcome. It has scenarios with windowed stores

Re: Kafka Streams Failed to rebalance error

Hi, Which version of Kafka are you using? This should be fixed in 0.10.2.1, any chance you could try that release? Thanks Eno > On 3 May 2017, at 14:04, Sameer Kumar wrote: > > Hi, > > > I ran two nodes in my streams compute cluster, they were running fine for few

Re: Kafka Stream stops polling new messages

; finished. > > Joao > > On Wed, May 3, 2017 at 6:22 AM Eno Thereska <eno.there...@gmail.com> wrote: > >> Hi there, >> >> Thanks for double checking. Does RocksDB actually crash or produce a crash >> dump? I’m curious how you know that the issue is

Re: Setting up Kafka & Kafka Streams for loading real-time and 'older' data concurrently

Just to add to this, there is a JIRA that tracks the fact that we don’t have an in-memory windowed store. https://issues.apache.org/jira/browse/KAFKA-4730 Eno > On May 3, 2017, at 12:42 PM, Damian Guy wrote: > > The

Re: Kafka Stream stops polling new messages

Hi there, Thanks for double checking. Does RocksDB actually crash or produce a crash dump? I’m curious how you know that the issue is https://github.com/facebook/rocksdb/issues/1121 , so just double checking with you. If that’s indeed the

Re: session window bug not fixed in 0.10.2.1?

Hi Ara, The PR https://github.com/apache/kafka/pull/2645 has gone to both trunk and 0.10.2.1, I just checked. What error are you seeing, could you give us an update? Thanks Eno On Fri, Apr 28, 2017 at 7:10 PM, Ara Ebrahimi wrote: > Hi, > > I upgraded to 0.10.2.1

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

; > topic - "20170502_instancea_1234" > brokers - "localhost:9092,localhost:9093,localhost:9094" > zookeepers - "localhost:2181,localhost:2182,localhost:2183" > maxMessageBytes - 3000 > ssl - true > > Thanks, > Henry > -- > Henry Thacker >

Re: Failure on timestamp extraction for kafka streams 0.10.2.0

Hi Sachin, This should be fixed in 0.10.2.1, could you upgrade to that release? Here is JIRA: https://issues.apache.org/jira/browse/KAFKA-4861 . Thanks Eno > On May 2, 2017, at 8:43 AM, Sachin Mittal wrote: > > The

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

Hi Henry, Could you share the streams configuration for your apps? I.e., the part where you assign application id and all the rest of the configs (just configs, not code). Thanks Eno > On May 2, 2017, at 8:53 AM, Henry Thacker wrote: > > Thanks all for your replies -

Re: Kafka-streams process stopped processing messages

rsion 0.10.1.1 > After a LONG startup time (more than an hour) where the status of the group > was rebalancing, all the 3 processes started processing messages again. > > This all thing started after we hit a bug in our code (NPE) that crashed > the stream processing thread. >

Re: Kafka-streams process stopped processing messages

Hi Shimi, Could you provide more info on your setup? How many kafka streams processes do you have and from how many partitions are they consuming from. If you have more processes than partitions some of the processes will be idle and won’t do anything. Eno > On Apr 30, 2017, at 5:58 PM, Shimi

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

"mystore").withStringKeys().withByteArrayValues().persistent().build(); >> >> KStream<Bytes, Bytes> stream = builder.stream(topic); >> >> builder.addStateStore(stateStore); >> >> stream.process(this::buildStreamProcessor, "mystore"); >> >&

Re: Kafka Streams 0.10.0.1 - multiple consumers not receiving messages

Hi Henry, Could you share the code that builds your topology so we see how the topics are passed in? Also, this would depend on what the streaming logic is doing with the topics, e.g., if you're joining them then both partitions need to be consumed by the same instance. Eno > On 28 Apr 2017,

Re: Stream applications dying on broker ISR change

e application running in the hope it would recover; After > ~1hour it's still just continuously spilling out the same exception and not > managing to continue processing. > > On 25 April 2017 at 16:24, Eno Thereska <eno.there...@gmail.com> wrote: > >> Hi Ian, >&g

Re: Stream applications dying on broker ISR change

d. runLoop(StreamThread.java:592) at org.apache.kafka.streams.processor.internals.StreamThread. run(StreamThread.java:361) On 24 April 2017 at 16:02, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Sachin, > > In KIP-62 a background heartbeat thread was introduced to deal with t

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congrats! Eno > On Apr 25, 2017, at 12:17 PM, Rajini Sivaram wrote: > > Thanks everyone! > > It has been a pleasure working with all of you in the Kafka community. Many > thanks to the PMC for this exciting opportunity. > > Regards, > > Rajini > > On Tue, Apr 25,

Re: Stream applications dying on broker ISR change

d has died or say we simply stopped an > instance and needs to be booted out of the group. > > Thanks > Sachin > > > On Mon, Apr 24, 2017 at 5:55 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > >> Hi Ian, >> >> >> This is now fixed in 0

Re: Stream applications dying on broker ISR change

Hi Ian, This is now fixed in 0.10.2.1. The default configuration need tweaking. If you can't pick that up (it's currently being voted), make sure you have these two parameters set as follows in your streams config: final Properties props = new Properties(); ...

Re: [VOTE] 0.10.2.1 RC3

+1 tested the usual streams tests as before. Thanks Eno > On 21 Apr 2017, at 17:56, Gwen Shapira wrote: > > Hello Kafka users, developers, friends, romans, countrypersons, > > This is the fourth (!) candidate for release of Apache Kafka 0.10.2.1. > > It is a bug fix

Re: Calculating time elapsed using event start / stop notification messages

Hi Ali, One starting point would be the low level Processor API, where you get each event and process it. You can also use a persistent state store to keep track of the events seen so far, it can probably be an in-memory store. An an entry can probably be deleted once both start and stop

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

tances(AbstractConfig.java:254) >at org.apache.kafka.common.config.AbstractConfig. > getConfiguredInstances(AbstractConfig.java:220) >at org.apache.kafka.clients.consumer.KafkaConsumer.( > KafkaConsumer.java:673) >... 6 more > > > > On Tue, Apr 18, 2017

Re: Kafka Streams - Join synchronization issue

; > And MyStreamUtils::locationActivityJoiner does: > > public static JsonObject locationActivityJoiner(JsonObject activity, String > loc) { > JsonObject join = activity.copy(); > join.put("city" , loc); > return join; > } > > hum... your que

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

the jar published somewhere on github or as a part of > build pipeline? > > On Mon, Apr 17, 2017 at 9:18 PM, Eno Thereska <eno.there...@gmail.com> > wrote: > >> Not yet, but as soon as 0.10.2 is voted it should be. Hopefully this week. >> >> Eno >>> On 1

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

Not yet, but as soon as 0.10.2 is voted it should be. Hopefully this week. Eno > On 17 Apr 2017, at 13:25, Mahendra Kariya <mahendra.kar...@go-jek.com> wrote: > > Are the bug fix releases published to Maven central repo? > > On Sat, Apr 1, 2017 at 12:26 PM, Eno Thereska &l

Re: Kafka-Streams: Cogroup

Hi Kyle, (cc-ing user list as well) This could be an interesting scenario. Two things to help us think through it some more: 1) it seems you attached a figure, but I cannot seem to open it. 2) what about using the low level processor API instead of the DSL as approach 3? Do you have any

Re: [VOTE] 0.10.2.1 RC1

+1 (non-binding) Built sources, ran all unit and integration tests, checked new documentation, esp with an eye on the streams library. Thanks Gwen Eno > On 12 Apr 2017, at 17:25, Gwen Shapira wrote: > > Hello Kafka users, developers, client-developers, friends, romans, >

Re: Kafka Streams Application does not start after 10.1 to 10.2 update if topics need to be auto-created

No, internal topics do not need to be manually created. Eno > On 13 Apr 2017, at 10:00, Shimi Kiviti wrote: > > Is that (manual topic creation) also true for internal topics? > > On Thu, 13 Apr 2017 at 19:14 Matthias J. Sax wrote: > >> Hi, >> >>

Re: Streams error handling

Hi Mike, Thank you. Could you open a JIRA to capture this specific problem (a copy-paste would suffice)? Alternatively we can open it, up to you. Thanks Eno > On 13 Apr 2017, at 08:43, Mike Gould wrote: > > Great to know I've not gone off in the wrong direction > Thanks

Re: In kafka streams consumer seems to hang while retrieving the offsets

ize by more aggressive compaction and let me see if that > will make things better. > > I will try to get the logs when this happens next time. > > Thanks > Sachin > > > > On Sun, Apr 9, 2017 at 6:05 PM, Eno Thereska <eno.there...@gmail.com> wrote: > >&

  1   2   3   >