RE: How to always consume from latest offset in kafka-streams

2018-01-22 Thread TSANG, Brilly
If you are doing dynamic assignment ( consumer.subscription), you can try this in your code: KafkaConsumer consumer = new KafkaConsumer<>(props); consumer.subscribe(Collections.singletonList("your_topic"), this); consumer.poll(0) //Just so you are connected and will have TopicPartition dynam

RE: can't feed remote broker with producer demo

2018-01-22 Thread TSANG, Brilly
Hi Rotem, I'm not 100% sure but you can try set listeners in \config\server.properties and see if that help. Regards, Brilly -Original Message- From: Rotem Jacobi [mailto:rot...@radcom.com] Sent: Monday, January 22, 2018 11:16 PM To: users@kafka.apache.org Subject: can't feed remote bro

can't feed remote broker with producer demo

2018-01-22 Thread Rotem Jacobi
Hi, When running the quickstart guide (producer, broker (with zookeeper) and consumer on the same machine) it works perfectly. When trying to run the producer from another machine to feed the broker I'm getting an error message: C:\Development\kafka_2.11-0.11.0.0\bin\windows>kafka-console-produc

Re: How to always consume from latest offset in kafka-streams

2018-01-22 Thread Xin Li
This? consumer.auto.offset.reset = latest Best, Xin On 19.01.18, 19:34, "Saloni Vithalani" wrote: Our requirement is such that if a kafka-stream app is consuming a partition, it should start it's consumption from latest offset of that partition. This seems like do-able usin

Re: offsetsForTimes API performance

2018-01-22 Thread Andrew Otto
Speaking of, has there been any talk of combining those two requests into a single API call? I’d assume that offsetForTimes + consumer seek is probably the most common use case of offsetForTimes. Maybe a round trip could be avoided if the broker could just auto-assign the consumer to the offset f

offsetsForTimes API performance

2018-01-22 Thread srimugunthan dhandapani
Hi all, We use kafka as our store and every one of our record is associated with a timeStamp. We pull data from kafka by seeking to a timeStamp offset everytime and then get the records by polling. We use KafkaConsumer's offsetsForTimes ( https://kafka.apache.org/0110/javadoc/org/apache/kafka/cl

Re: Does Kafka consumer api provide a way for you to search for partition number of a topic with key?

2018-01-22 Thread Brilly Tsang
Many thanks Andras!!! That's exactly what I am looking for. Regards, Brilly ⁣Sent from TypeApp ​ On Jan 22, 2018, 21:37, at 21:37, Andras Beni wrote: >Hi Brilly, > >Do I understand right, that you want your consumers to consume only >from a >given partition / partitions based on a preconfigured

Re: Kafka functionality

2018-01-22 Thread RedShift
On 21/01/2018 10:15, Aman Shaikh wrote: Dear sir/madam. Can we use Kafka for handling streams of data from gps device coming per sec from 1000's of device. Thanks and Regards, Aman Yes.

Re: Does Kafka consumer api provide a way for you to search for partition number of a topic with key?

2018-01-22 Thread Andras Beni
Hi Brilly, Do I understand right, that you want your consumers to consume only from a given partition / partitions based on a preconfigured value? You need to copy the mechanism the default partitioner (if this is used on producer side) uses to determine partition. ( https://github.com/apache/kafk