I need some help with the production server architecture

Hi, Sometime back i was informed on the group that in production we should never run kafka on same physical machine. So based on that I have a question on how to divide the server nodes we have to run zookeper and kafka brokers. I have a following setup Data center 1 Lan 1 (3 VMs) 192.168.xx.yy1

Question about state store

Hello all, I'm not sure how to do the following use-case, if anyone can help :) - I have an admin UI where the admin can choose wich item are allowed on the application - When the admin choose an item, it pushes an object to a topic kafka : test.admin.item with the following object

Two possible issues with 2.11 0.10.1.0

Ran into these two internal exceptions today. Likely they are known or my fault but I'm reporting them anyway: Exception in thread "StreamThread-1" java.lang.NullPointerException at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:304) at

Re: while publishing message need to add multiple keys in a single message

Kafka doesn't have the concept of message headers like some other messaging systems. You will have to create a payload that contains these headers and whatever bytes you are sending. Dave > On Nov 28, 2016, at 16:47, Prasad Dls wrote: > > Hi, > > While publishing each

HOW TO GET KAFKA CURRENT TOPIC GROUP OFFSET

HI! WAIT YOUR ANSWER

Re: Kafka Streaming

I just cloned 3.1x and tried to run a test. I am still seeing rocksdb error: Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError: C:\Users\manchlia\AppData\Local\Temp\librocksdbjni108789031344273.dll: Can't find dependent libraries On Mon, Oct 24, 2016 at 11:26 AM, Matthias

Re: Kafka Connect consumer not using the group.id provided in connect-distributed.properties

Hi, I am using Kafka Connect Sink Connector to consume messages from a Kafka topic in a secure Kafka cluster. I have provided the group.id in connect-distributed.properties. I am using security.protocol as SASL_PLAINTEXT. Here is definition of group id in connect-distributed.properties -

Kafka Connect consumer not using the group.id provided in connect-distributed.properties

Hi, I am using Kafka Connect Sink Connector to consume messages from a Kafka topic in a secure Kafka cluster. I have provided the group.id in connect-distributed.properties. I am using security.protocol as SASL_PLAINTEXT. Here is definition of group id in connect-distributed.properties -

Re: Initializing StateStores takes *really* long for large datasets

I'll write an update on where I am now. I've got about 40 'primary' topics, some small, some up to about 10M messages, and about 30 internal topics, divided over 6 stream instances, all running in a single app, talking to a 3 node Kafka cluster. I use a single thread per stream instance, as my

Kafka Clients Survey

Hey Kafka Community, I'm trying to take a pulse on the current state of the Kafka clients ecosystem. Which languages are most popular in our community? What does the community value in clients? You can help me out by filling in the survey: https://goo.gl/forms/cZg1CJyf1PuqivTg2 I will lock the

while publishing message need to add multiple keys in a single message

Hi, While publishing each message (single message) to Kafka, I need to add below headers/key ID: 123456 TYPE: xyz EVENTE: A OPERATION: Insert createTimeStampt: 2016-11-24T19:41:23.354Z updatedTimeStamp: 2016-11-30T19:41:23.354Z Please help me on this, how can i add all these into single

Re: kafka 0.10.1 ProcessorTopologyTestDriver issue with .map and .groupByKey.count

Hamid, would you mind creating a Jira? Thanks. -Matthias On 11/28/16 9:36 AM, Guozhang Wang wrote: > Damian, Hamid: > > I looked at the source code and suspect that it is because of the > auto-repartitioning which causes the topology to not directly forward the > record to the child

Re: Kafka 0.10.1.0 consumer group rebalance broken?

There has been plenty of changes in the GroupCoordinator and co between these two releases: https://github.com/apache/kafka/commit/40b1dd3f495a59abef8a0cba5450526994c92c04#diff-96e4cf31cd54def6b2fb3f7a118c1db3 It might be related to this:

Kafka 0.10.1.0 consumer group rebalance broken?

Hi, It seems that consumer group rebalance is broken in Kafka 0.10.1.0 ? When running a small test-project : - consumers running in own JVM (with different 'client.id') - producer running in own JVM - kafka broker : the embedded kafka : KafkaServerStartable It looks like the consumers loose

Re: Initializing StateStores takes *really* long for large datasets

Hello Frank, How many instances do you have in your apps and how many threads did you use per thread? Note that besides the topology complexity (i.e. number of state stores, number of internal topics etc) the (re-)initialization process is depending on the underlying consumer's membership

Re: KafkaStream: puncutuate() never called even when data is received by process()

Yes we are considering to differentiate "process" and "punctuate" function to be "data-driven" and "time-driven" computations. That is, triggering of punctuate should NOT be depending on the arrival of messages, or the message's associated timestamps. As for now I think periodically inserting the

Problem consuming using 0.10.1.0

(caveat - was having this issue with 0.10.0 but thought an update might help) App has a simple layout: consume -> aggregate -> output to 2nd topic Starting the app I see pages of this sort of message: DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0xb58abf2daaf0002

RE: Kafka consumers are not equally distributed

Thank you, Guozhang. I see a lot of this exception: org.apache.kafka.clients.consumer.internals.ConsumerCoordinator::Offset commit failed. : TimeoutException: The request timed out. We are committing the offset manually by Asynch mode and session.timeout.ms is 5 mins and poll time 10 secs and

Re: Kafka windowed table not aggregating correctly

Sachin, This is indeed a bit wired, and we'd like to try to re-produce your issue locally. Do you have a sample input data for us to try out? Guozhang On Fri, Nov 25, 2016 at 10:12 PM, Sachin Mittal wrote: > Hi, > I fixed that sorted set issue but I am facing a weird

Re: Interactive Queries

Thanks All. On Mon, Nov 28, 2016 at 3:09 AM, Michael Noll wrote: > There are also some examples/demo applications at > https://github.com/confluentinc/examples that demonstrate the use of > interactive queries: > > - > https://github.com/confluentinc/examples/blob/3. >

Re: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

I can’t see the code, I think it’s not added to the email? Thanks Eno From: Hamza HACHANI Reply-To: Date: Monday, 28 November 2016 at 13:25 To: "users@kafka.apache.org" Subject: RE: Abnormal working in the method

Re: Messages intermittently get lost

Thanks Martin, I will look at those links. But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper? From: Martin Gainty

RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

the print is in line 40 of the class Base... De : Hamza HACHANI Envoyé : lundi 28 novembre 2016 01:25:08 À : users@kafka.apache.org Objet : RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms Hi Eno,

RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Eno, Here is the code for the application ExclusiveStatsConnectionDevice which is composed of 4 nodes. For example when i put print("") you would sess the problem of the infinite loop. I preferred to send the whole code sto make it easier to you even though you don't need all of it

Re: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Hamza, Would you be willing to share some of your code so we can have a look? Thanks Eno > On 28 Nov 2016, at 12:58, Hamza HACHANI wrote: > > Hi Eno. > > The problem is that there is no infinite while loop that i write. > > So I can't understand why the

RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Eno. The problem is that there is no infinite while loop that i write. So I can't understand why the application is doing so. Hamza De : Eno Thereska Envoyé : dimanche 27 novembre 2016 23:21:24 À : users@kafka.apache.org Objet : Re:

Re: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Hamza, If you have an infinite while loop, that would mean the app would spend all the time in that loop and poll() would never be called. Eno > On 28 Nov 2016, at 10:49, Hamza HACHANI wrote: > > Hi, > > I've some troubles with the method puctuate.In fact when

Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi, I've some troubles with the method puctuate.In fact when i would like to print a string in the method punctuate. this string would be printed in an indefinitly way as if I printed (while (true){print(string)}. I can't understand what happened.Does any body has an explenation ?. Besides

Re: 0.8 consumers compatibility with Kafka 0.10

Hi Vladi, Generally, newer brokers are compatible with older clients. As Vladimir said, the upgrade page (http://kafka.apache.org/documentation.html#upgrade) includes detailed information about changes that could have a compatibility impact. Ismael On Sun, Nov 27, 2016 at 8:09 PM, Vladi Feigin

Re: Interactive Queries

There are also some examples/demo applications at https://github.com/confluentinc/examples that demonstrate the use of interactive queries: -