Re: Error while starting kafka server in Windows

looks like you are running kafka on Windows OS. There may be some issues running kafka on windows, not sure. you can try on Linux. On Wed, Mar 22, 2017 at 9:24 AM, vani reddy wrote: > Hi, I am getting below error:I even restarted my system, but still the > same error. > Iam using the version k

Re: Consumers not rebalancing after replication

I would like to add one more thing. The logs on the consumers looks fine. We see no errors whatsoever. The log level is set INFO. What are we missing here? Is there some other config that we need to set so that we at least see some errors for this in the logs? On Wed, Mar 22, 2017 at 9:14 AM, Mahe

Kafka Connect behaves weird in case of zombie Kafka brokers. Also, zombie brokers?

Hello everyone, We are running a 5 broker Kafka v0.10.0.0 cluster on AWS. Also, the connect api is in v0.10.0.0. It was observed that the distributed kafka connector went into infinite loop of log message of (Re-)joining group connect-connect-elasticsearch-indexer. And after a little more digging

Error while starting kafka server in Windows

Hi, I am getting below error:I even restarted my system, but still the same error. Iam using the version kafka_2.11-0.10.1.0 java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section o at java.io.RandomAccessFile.setLength(Native Method)

Consumers not rebalancing after replication

Hey All, We have six consumers in a consumer group. At times, some of the partitions are under replicated for a while (maybe, 2 mis). During this time, the consumers subscribed to such partitions stops getting data from Kafka and they become inactive for a while. But when the partitions are fully

Re: Question about kafka-streams task load balancing

Hi, I guess, it's currently not possible to load balance between different machines. It might be a nice optimization to add into Streams though. Right now, you should reduce the number of threads. Load balancing is based on threads, and thus, if Streams place tasks to all threads of one machine,

Re: Are there Connector artifacts in Confluent or any other Maven repository?

Yes, these get published to Confluent's maven repository. Follow the instructions here http://docs.confluent.io/current/installation.html#installation-maven for adding the Confluent maven repository to your project and then add a dependency for the connector to your project (e.g. for that partition

Question about kafka-streams task load balancing

Hey, I have a typical scenario of a kafka-streams application in a production environment. We have a kafka-cluster with multiple topics. Messages from one topic is being consumed by a the kafka-streams application. The topic, currently, has 9 partitions. We have configured consumer thread coun

Re: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Just to clarify, I did want to have the term `Topology` as part of the class name, for the reasons above. I'm not too worried about to be consistent with the previous names, but I feel the `XXTopologyBuilder` is better than `XXStreamsBuilder` since it's build() function returns a Topology object.

Are there Connector artifacts in Confluent or any other Maven repository?

I am trying to migrate from StreamX (https://github.com/qubole/streamx) to use the official Confluent S3 connector (https://github.com/confluentinc/kafka-connect-storage-cloud). Part of my implementation of Kafka Connect requires a custom partitioner. This partitioner originally extended the

Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

Hi Michael, Thank you very much for the prompt response, really appreciate it! >From https://github.com/confluentinc/examples/blob/3.2.x/kafka- streams/src/main/java/io/confluent/examples/streams/Wor dCountLambdaExample.java#L55-L62 and https://github.com/confluentinc/examples/tree/3.2.x/kafka-st

Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

Typically you'd containerize your app and then launch e.g. 10 containers if you need to run 10 instances of your app. Also, what do you mean by "in a cluster of Kafka containers" and "in the cluster of Kafkas"? On Tue, Mar 21, 2017 at 9:08 PM, Mina Aslani wrote: > Hi, > > I am trying to underst

Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

Hi, I am trying to understand how I can use a kafka stream app(jar file) in a cluster of kafka containers. Kafka does not have master/slave concept (unlike spark), how I should run my app in the cluster of kafkas (e.g. on one or multiple docker-machine/s)? I use below command line when having on

Re: Commits of slow moving topics in Kafka Streams time out

Hi Frank, There is a similar discussion here https://www.mail-archive.com/users@kafka.apache.org/msg25089.html , with a JIRA. Have you tried to increase the retention minutes to something large? Thanks Eno > On 21 Mar 2017, at

Commits of slow moving topics in Kafka Streams time out

Hi Kafka people, We have a Kafka Streams application that replicates a database, and transforms it to a different data model. Some tables/topics move fast, with many changes a second, some might be dormant for months. For those slow moving topics, we have some trouble with the 'offsets.retention.m

Re: Out of order message processing with Kafka Streams

Hans, Which class's javadocs should i look at? From my initial look at the javadocs and discussion with Michael, it doesn't seem possible. On Tue, Mar 21, 2017 at 10:44 PM, Hans Jespersen wrote: > Yes, and yes! > > -hans > > > > > On Mar 21, 2017, at 7:45 AM, Ali Akhtar wrote: > > > > That wou

Re: Out of order message processing with Kafka Streams

Yes, and yes! -hans > On Mar 21, 2017, at 7:45 AM, Ali Akhtar wrote: > > That would require > > - Knowing the current window's id (or some other identifier) to > differentiate it from other windows > > - Being able to process individual messages in a window > > Are those 2 things possible

Re: Streams RocksDBException with no message?

Thanks Guozhang. For my part, turns out I was hitting ulimit on my open file descriptors. Phew, easy to fix... once I figured it out. :-) Mathieu On Fri, Mar 17, 2017 at 4:14 PM, Guozhang Wang wrote: > Hi Mathieu, > > We are aware of that since long time ago and I have been looking into this

Re: Out of order message processing with Kafka Streams

Hi Ali, (My use case is, i receive a stream of messages. Messages need to be stored > and sorted into 'buckets', to indicate 'sessions'. Each time there's a gap > of 30 mins or more since the last message (under a key), a new 'session' > (bucket) should be started, and future messages should belon

Re: Kafka consumer offset reset

What offset do you want to reset them to? The easier way to adjust offsets in 0.10 is to attach a consumer for the target topic-partition and seek to the position you desire and commit that new offset. On Tue, Mar 21, 2017 at 9:56 AM, Jakub Stransky wrote: > Hello, > > just recently migrated to

Kafka consumer offset reset

Hello, just recently migrated to using Kafka 0.10.1.0 I would like to reset position for some consumers. I went through documentation and couldn't spot it how to achieve that. All what I got is that v 10 reduces usage of zookeeper and clients have possibility to use different storage for maintain

Re: Out of order message processing with Kafka Streams

That would require - Knowing the current window's id (or some other identifier) to differentiate it from other windows - Being able to process individual messages in a window Are those 2 things possible w/ kafka streams? (java) On Tue, Mar 21, 2017 at 7:43 PM, Hans Jespersen wrote: > While it

Re: Out of order message processing with Kafka Streams

While it's not exactly the same as the window start/stop time you can store (in the state store) the earliest and latest timestamps of any messages in each window and use that as a good approximation for the window boundary times. -hans > On Mar 20, 2017, at 1:00 PM, Ali Akhtar wrote: > > Y