Re: kafka consumer rebalance

> of migration of old consumers > to new consumers is not available. More details here: > https://issues.apache.org/jira/browse/KAFKA-4513 > > On Wed, Aug 30, 2017 at 11:05 AM, sunil kalva <kalva.ka...@gmail.com> > wrote: > > > Hi > > Can someone help me he

Re: kafka consumer rebalance

Hi Can someone help me here ..! On Tue, Aug 29, 2017 at 11:57 PM, sunil kalva <kalva.ka...@gmail.com> wrote: > Hi > Is there any change in consumer rebalancing algorithm from 0.8.2 and > 0.10.2 versions. If yes how do i do rolling upgrades of consumer client > applications, w

kafka consumer rebalance

Hi Is there any change in consumer rebalancing algorithm from 0.8.2 and 0.10.2 versions. If yes how do i do rolling upgrades of consumer client applications, while doing rolling upgrade few of clients might be with older algo and few might be with newer algo then rebalancing might not be working

consumer rebalancing (0.8.2.2) not happening

*I read this from kafka docs (https://github.com/SOHU-Co/kafka-node/issues/124 ), "When rebalancing is triggered at one consumer, rebalancing should be triggered in other consumers", I am emitting rebalance metrics for jmx bean

Re: Removing kafka topic contents from Java

Are you looking something like this String command = String.format("--delete --zookeeper %s --topic %s", "localhost:2181", "test_topic"); String[] split = command.split(" "); TopicCommand.main(split); t SunilKalva On Mon, Oct 24, 2016 at 9:09 PM, Ali Akhtar wrote: >

Re: delete topic causing spikes in fetch/metadata requests

Hi Can you guys help me with this issue On Oct 12, 2016 10:35 PM, "sunil kalva" <sambarc...@gmail.com> wrote: > > We are using kafka 0.8.2.2 (client and server), when ever we delete a > topic we see lot of errors in broker logs like below, and there is also a > spik

delete topic causing spikes in fetch/metadata requests

We are using kafka 0.8.2.2 (client and server), when ever we delete a topic we see lot of errors in broker logs like below, and there is also a spike in fetch/metadata requests. Can i correlate these errors with topic delete or its a known issue. Since there is spike in metadata requests and fetch

NotLeaderForPartitionException causing event loss

We are seeing data loss, whenever we see "NotLeaderForPartitionException" exception, We are using 0.8.2 java client publisher API with callback, when i get callback with the error i am logging them in a file and retrying them later. So number of errors = number of logged events are matching but

Re: Messages corrupted in kafka

messages, (< 0.1%). If so are you > able to see some corrupted messages if you produce, say, 10M messages? > > On Wed, Mar 23, 2016 at 9:40 PM, sunil kalva <kalva.ka...@gmail.com> > wrote: > > > I am using java client and kafka 0.8.2, since events are corrupted in >

Re: Messages corrupted in kafka

(Becket) Qin > > On Wed, Mar 23, 2016 at 8:28 PM, sunil kalva <kalva.ka...@gmail.com> > wrote: > > > can some one help me out here. > > > > On Wed, Mar 23, 2016 at 7:36 PM, sunil kalva <kalva.ka...@gmail.com> > > wrote: > > > > > Hi &

Re: Messages corrupted in kafka

can some one help me out here. On Wed, Mar 23, 2016 at 7:36 PM, sunil kalva <kalva.ka...@gmail.com> wrote: > Hi > I am seeing few messages getting corrupted in kafka, It is not happening > frequently and percentage is also very very less (less than 0.1%). > > Basically i

Messages corrupted in kafka

Hi I am seeing few messages getting corrupted in kafka, It is not happening frequently and percentage is also very very less (less than 0.1%). Basically i am publishing thrift events in byte array format to kafka topics(with out encoding like base64), and i also see more events than i publish (i

retrieve commit time for messages

Hi Is there any way to get the commit timestamp of the messages which are retrieved using kafka consumer API. t SunilKalva

Re: replication factor increased automatically !

t; > Topic: part_2_repl_3 Partition: 1 Leader: 6 Replicas: 6,4,7 Isr: 7,4,6 > > > The line in bold (i.e. the first line in the output of the kafka-topics.sh) > gives on a topic level - #partitions, #replicationfactor etc > Please paste the full output > &

Re: replication factor increased automatically !

llo Sunil, > Are you able to see 8 replicas(i.e. in ISR) for all the partitions? > > Can you paste the output of *--describe *topic command? > kafka-topics --zookeeper *-describe* > > On Mon, Nov 23, 2015 at 4:29 PM, sunil kalva <kalva.ka...@gmail.com> > wrote:

Re: replication factor increased automatically !

We found the issue, it was with our UI tool which had a bug in partition reassignment tool. Nothing todo with kafka. Sorry for spamming. t Sunil On Mon, Nov 23, 2015 at 4:13 PM, sunil kalva <kalva.ka...@gmail.com> wrote: > hi > sorry missed the header in my prev mail > > kafk

replication factor increased automatically !

Hi We have a 10 node kafka cluster, and we have created a topic with replication factor "3" . It was running fine for 10 days, after that the replication factor is increased to "8" with out running any command to increase the replication. We have verified in server logs and also confirmed with the

kafka metrics emitting to graphite

How to configure, to emit kafka broker metrics to graphite. t SunilKalva

Re: producer hung after some time

We also checked the jconsole for memory graph, it was taking only 80% of the heap (i can see still 20% remaining). -SK On Tue, Sep 29, 2015 at 8:13 PM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > also check your'e GC, caused by using async without backpressuce, with the > latest

New Async producer hangs

Hi I am using new asyc java producer , after couple of minutes i am getting following error prio=5 os_prio=31 tid=0x7f9383685800 nid=0xea03 waiting for monitor entry [0x000123a6a000] java.lang.Thread.State: BLOCKED (on object monitor) at

Re: Error handling in New AsyncProducer

appropriately. On Mon, Aug 17, 2015 at 7:15 PM, sunil kalva kalva.ka...@gmail.com wrote: tx jeff, Actually we need to set buffer.memory to minimum (default is ~35 MB) and block.on.buffer.full to true so that the sender will block as soon as these conditions met. And then release once

Re: Error handling in New AsyncProducer

an Exception in callback if there is problem while sending data. You can check like: public void onCompletion(RecordMetadata arg0, Exception arg1) { if (arg1 != null) { System.out.println(exception occured); } System.out.println(sent) On Mon, Aug 17, 2015 at 9:20 PM, sunil kalva

Error handling in New AsyncProducer

Hi all I am using new java producer in async mode, when my entire cluster is down i am loosing all my messages. How do we get notification when the cluster is down so that i can send messages to another cluster. The callback is only triggered when the cluster is reachable . --SK

Fwd: Spooling support for kafka publishers !

-- Forwarded message -- From: sunil kalva kalva.ka...@gmail.com Date: Fri, Aug 7, 2015 at 2:12 PM Subject: Spooling support for kafka publishers ! To: d...@kafka.apache.org Hi What are the best practises to achieve spooling support on producer end if the kafka cluster

simple consumer giving error always

Hi For few topics i always get FetchResponse.error code as 1, i am using simple consumer 0.8.1. What are the reasons to get this error. and any fix for this ? -- SunilKalva

Re: running kafka/zookeeper running locally

could get away with mocking out the Kafka broker, depending on what/how you like to test. On Sun, Apr 19, 2015 at 10:47 PM, sunil kalva kalva.ka...@gmail.com wrote: Hi Any one tried running zookeeper and kafka locally , which can be useful for automating the test cases for API built

Re: running kafka/zookeeper running locally

...@gmail.com: Also consider with whether you could get away with mocking out the Kafka broker, depending on what/how you like to test. On Sun, Apr 19, 2015 at 10:47 PM, sunil kalva kalva.ka...@gmail.com wrote: Hi Any one tried running zookeeper and kafka locally , which can be useful

running kafka/zookeeper running locally

Hi Any one tried running zookeeper and kafka locally , which can be useful for automating the test cases for API built on kafka ? SunilKalva

Re: blocking KafkaProducer call

, Mar 31, 2015 at 11:21 PM, sunil kalva sambarc...@gmail.com wrote: thanks ghenke, that was a quick response. I will test and will let you know if i have some questions. On Tue, Mar 31, 2015 at 11:17 PM, Grant Henke ghe...@cloudera.com wrote: I think you are looking at is this section: If you

Re: blocking KafkaProducer call

On Wed, Apr 1, 2015 at 12:34 PM, sunil kalva sambarc...@gmail.com wrote: Does these config params has effect when i try to simulate sync mode by not passing callback ? On Wed, Apr 1, 2015 at 10:32 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Whats your linger.ms and batch.size

Re: blocking KafkaProducer call

Does these config params has effect when i try to simulate sync mode by not passing callback ? On Wed, Apr 1, 2015 at 10:32 PM, Mayuresh Gharat gharatmayures...@gmail.com wrote: Whats your linger.ms and batch.size ? Thanks, Mayuresh On Wed, Apr 1, 2015 at 5:51 AM, sunil kalva sambarc

blocking KafkaProducer call

Hi According to this http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html documentation, if i don't pass callback it will work as blocking call, Does it mean that message will be immediately sent to kafka cluster and all possible exceptions will be

Re: blocking KafkaProducer call

: http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html#get() On Tue, Mar 31, 2015 at 12:30 PM, sunil kalva sambarc...@gmail.com wrote: Hi According to this http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html

kafka audit

Hi What is best practice for adding audit feature in kafka, Is there any framework available for enabling audit feature at producer and consumer level and any UI frameworks for monitoring. tx SunilKalva

Re: New Java Producer Client handling case where Kafka is unreachable

20, 2015 at 3:36 PM, sunil kalva sambarc...@gmail.com wrote: I think KafkaProducer.send method blocks until it fetches partition metadata for configured time using metadata.fetch.timeout.ms, once time out it throws TimeoutException. You might be experiencing TimeoutException ? My co-worker

Re: New Java Producer Client handling case where Kafka is unreachable

I think KafkaProducer.send method blocks until it fetches partition metadata for configured time using metadata.fetch.timeout.ms, once time out it throws TimeoutException. You might be experiencing TimeoutException ? ref: KafkaProducer.java(waitOnMetadata) On Fri, Mar 20, 2015 at 2:42 PM, Samuel

Re: publisher spooling ....!

limited persistence to the producer. You can see it here: https://issues.apache.org/jira/browse/KAFKA-1955 -Jay On Tue, Mar 3, 2015 at 8:15 AM, sunil kalva kalva.ka...@gmail.com wrote: Hi Is there any way to spool messages to disk at publisher side when kafka cluster is down or not reachable

Re: Async Producer Callback

:05 AM, sunil kalva sambarc...@gmail.com wrote: future returns RecordMetadata class which contains only metadata not the actual message. But i think *steven* had a point like saving the reference in impl class and retry if there is an exception in callback method. On Thu, Mar 19, 2015

producer queue size

Hi How do i get the size of the inmemory queue which are holding messages and ready to send in async producer, i am using new KafkaProducer class in 0.8.2. Basically instead of dropping the messages silently, i want to avoid sending messages if the queue is already full. I am using async

Re: producer queue size

essentially i want to use this property queue.buffering.max.messages with new KafkaProducer class, and also want to access the current value of the queue SunilKalva On Wed, Mar 18, 2015 at 9:51 PM, sunil kalva sambarc...@gmail.com wrote: Hi How do i get the size of the inmemory queue which

Async Producer Callback

Hi How do i access the actual message which is failed to send to cluster using Callback interface and onCompletion method. Basically if the sender is failed to send, i want to add it to a temp queue and retry them later. --Sunil

Re: High level consumer group

, you need to set a different group.id for each. On Mon, Mar 16, 2015 at 10:10 AM, sunil kalva sambarc...@gmail.com wrote: How do i deploy high level consumer group for one topic in multiple instances of consumer applications, i mean i have two consumer applications deployed in two

Re: kafka mirroring ...!

to a different topic name in the current mirror maker implementation. Mirror maker sends the message to destination topic based on the topic name it reads from source On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva sambarc...@gmail.com wrote: Can i configure different topic name in destination cluster

Re: kafka mirroring ...!

And i also observed ,all the data is moving to one partition in destination cluster though i have multiple partitions for that topic in source and destination clusters. SunilKalva On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva sambarc...@gmail.com wrote: I ran kafka mirroring tool after producing

kafka mirroring ...!

I ran kafka mirroring tool after producing data in source cluster, and this is not copied to destination cluster. If i produce data after running tool those data are copied to destination cluster. Am i missing something ? -- SunilKalva

mapping between disk and partition

Hi Can i map a specific partition to a different disk in a broker. And what is the general recommendations for disk to partition mapping for which that broker is leader. and also for replications that broker handles. -- SunilKalva

publisher spooling ....!

://issues.apache.org/jira/browse/KAFKA-156, Is there any plan to support this in future or any alternatives. What is broker replication some one mentioned in the same jira, is there any documentation for that ? please advise me on this, my entire design depends on this feature. t Sunil Kalva

Re: moving replications

Is there any way to automate On Mar 3, 2015 11:57 AM, sunil kalva sambarc...@gmail.com wrote: Why can't kafka automatically rebalances partitions with new broker and adjust with existing brokers ? Why should we run manually ? On Tue, Mar 3, 2015 at 6:41 AM, Gwen Shapira gshap...@cloudera.com

Re: moving replications

specified by the --broker-list option. The format to use is... So it sounds like what you looked for? Since the tool runs in dry run mode, you can try and see without actually moving anything. Gwen On Mar 2, 2015 10:05 AM, sunil kalva sambarc...@gmail.com wrote: Shapira tx for quick

load balancing

Is kafka load balancing based on number of partitions of a topic or number partitions of all topics in a cluster ? -- SunilKalva

moving replications

Hi How to move replications from one broker to another broker ? -- SunilKalva

Re: moving replications

/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool Its a bit tricky to use, so feel free to follow up with more questions :) Gwen On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva sambarc...@gmail.com wrote: Hi How to move replications from one broker to another broker

question related to offset reset

Can i do a offset reset based on time not by message id, since most of times we know when the processing engine failed (based on time), so it is easy to reset the offset to that time for reprocessing. -- SunilKalva

Custom partitioner in kafka-0.8.2.0

Hi I could not find a way to customize Partitioner class in new KafaProducer class, is it intentional ? tx SunilKalva

Re: Custom partitioner in kafka-0.8.2.0

Hi I could not find a way to customize Partitioner class in new KafaProducer class, is it intentional ? tx SunilKalva

Re: Custom partitioner in kafka-0.8.2.0

/082/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html ManiKumar On Thu, Feb 19, 2015 at 6:05 PM, sunil kalva kalva.ka...@gmail.com wrote: Hi I could not find a way to customize Partitioner class in new KafaProducer class, is it intentional ? tx SunilKalva