Re: How to reduce time for consumer to take over?

Hello Christian, Assuming that you are using new Java kafka-clients with Kafka's consumer group management, you'd have to tune consumers and broker. See - Broker config https://kafka.apache.org/documentation/#brokerconfigs -- group.initial.rebalance.delay.ms -- group.max.session.timeout.ms --

Re: Streams dependency on RocksDB - licenses

Created https://issues.apache.org/jira/browse/KAFKA-5977 for this issue. On Tue, Sep 26, 2017 at 9:49 AM, Stevo Slavić <ssla...@gmail.com> wrote: > There is legal problem for older rocksdb versions, see > https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=1610

Re: Streams dependency on RocksDB - licenses

SE > > > > The second commit you referenced, adds GPL2 to the list of licenses in > the > > pom, but it's not clear why. > > > > Ismael > > > > On Thu, Sep 21, 2017 at 3:44 PM, Stevo Slavić <ssla...@gmail.com> wrote: > > > >>

Re: How would Kafka behave in this scenario

Hello Sameer, Behavior depends on min.insync.replicas configured for the topic. Find more info in the documentation https://kafka.apache.org/documentation/#topicconfigs Kind regards, Stevo Slavic. On Tue, Sep 26, 2017 at 9:01 AM, Sameer Kumar wrote: > In case one of

Re: Streams dependency on RocksDB - licenses

/6e3ee015fb1ce03e47838e9a3995410ce884c212#diff-1d5a1a614aa158e14292a60d9891f1df Shouldn't similar notice be added to Apache Kafka too? On Thu, Sep 21, 2017 at 3:43 PM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello Apache Kafka community, > > Is it on purpose that kafka-streams 0

Streams dependency on RocksDB - licenses

Hello Apache Kafka community, Is it on purpose that kafka-streams 0.11.0.1 depends on org.rocksdb:rocksdbjni:5.0.1, and not on newer 5.7.3, because 5.0.1 has Apache 2 license while 5.7.3 has also GPL 2.0 parts? Kind regards, Stevo Slavic.

"client" and requests in request.timeout.ms

Hello Apache Kafka community, In Consumer, Producer, AdminClient and Broker configuration documentation there's a common config property, request.timeout.ms, with common description part being: "The configuration controls the maximum amount of time the client will wait for the response of a

Re: Message polling behavior when subscribed to a topic pattern

Hello William, You have overriden KafkaConsumer's max.poll.records configuration, to return only up to 10 records in single call to poll, so it does that. Fetcher iterates over completed fetch responses and in first one(s) from single topic finds enough of records to satisfy max limit. See

Re: Kafka consumer offset reset

Hello Jakub, Maybe you're interested in feature that's not yet available but is being proposed and discussed (on dev ML) for future - see https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling Kind regards, Stevo Slavic. On Tue, Mar 21, 2017 at

Re: Kafka client and ELB (round-robin DNS)

See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html On Fri, Mar 3, 2017, 20:57 BGCH wrote: > Hi all, > > briefly about the problem, if bootstrap.servers is a domain name and > that domain name eventually starts pointing to different

Re: Kafka partition no migrating to another broker.

Hello Shri, That behavior is by current Apache Kafka design. At topic creation time, for every topic partition, replication factor is converted to a replica set (set of ids of brokers which should replicate the partition), and those per partition replica sets is the metadata the gets stored in

Re: zookeeper.set.acl=false not considered by ZookeeperLeaderElector

on, Feb 27, 2017 at 3:03 PM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Hello Apache Kafka community, > > > > There's nice documentation on enabling ZooKeeper security on an existing > > Apache Kafka cluster at > > https://kafka.apache.org/documentation/#zk_a

zookeeper.set.acl=false not considered by ZookeeperLeaderElector

Hello Apache Kafka community, There's nice documentation on enabling ZooKeeper security on an existing Apache Kafka cluster at https://kafka.apache.org/documentation/#zk_authz_migration For your convenience here are the first two steps of migration: 1. Perform a rolling restart setting the

Re: Is there any command can be used to get broker.id on one of broker with command?

You can just grep it from broker's config/server.properties On Thu, Feb 16, 2017 at 7:42 AM, kay James wrote: > Is there any command can be used to get broker.id on one of broker with > command? > I can check zookeeper or check the broker starting log to find broker.id, >

Re: Fault tolerance not working in Kafka

Check __consumer-offsets topic, number of partitions and their replica assignment. You may be affected by problem solved by https://cwiki.apache.org/confluence/display/KAFKA/KIP-115%3A+Enforce+offsets.topic.replication.factor+upon+__consumer_offsets+auto+topic+creation On Mon, Feb 6, 2017, 06:32

Re: When publishing to non existing topic, TimeoutException is thrown instead of UnknownTopicOrPartitionException

Found somewhat related ticket https://issues.apache.org/ jira/browse/KAFKA-4385 On Fri, Jan 27, 2017 at 1:09 PM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello Apache Kafka community, > > When using new async KafkaProducer, one can register callback wi

When publishing to non existing topic, TimeoutException is thrown instead of UnknownTopicOrPartitionException

Hello Apache Kafka community, When using new async KafkaProducer, one can register callback with send calls. With auto.create.topics.enable set to false, when I try to publish to non existing topic, I expect callback to complete with UnknownTopicOrPartitionException. Instead, I get back

Re: Reg: Need info on Kafka Brokers

Hello Sumit, Not yet. AFAIK few topic management requests got introduced in broker side (see https://issues.apache.org/jira/browse/KAFKA-2229 ) but not yet in client APIs. Querying/listing topics request doesn't seem even to be planned yet. AdminUtils which talks with ZooKeeper via ZkClient is

Re: Unable to create new topics, "no brokers available"

Hello Alex, ZooKeeper nodes that Kafka brokers create in /brokers/ids path, upon registering themselves that they are available, are ephemeral, so not persistent. Ephemeral ZooKeeper nodes live as long as ZooKeeper session that created them is active. The /broker/ids child nodes are gone by

Common client metrics - per request?

Hello Apache Kafka community, Please correct me if wrong, I assume common Kafka client metrics (see https://kafka.apache.org/documentation.html#selector_monitoring ) are aggregated metrics of all different requests particular client instance makes. So e.g. producer common metrics like

Re: REST API to kafka topic

https://github.com/confluentinc/kafka-rest On Thu, Dec 15, 2016 at 8:12 PM, Gautham Kumaran < gautham.kumara...@gmail.com> wrote: > Hi Everyone, > > I'm trying to learn Kafka. So this might be basic :) > > I've come across the following API call to a kafka topic directly, but I > was under the

Release Kafka 0.9.0.2?

Hello Apache Kafka community, Would it be possible to release 0.9.0.2? It has few important fixes, like KAFKA-3594 and would be helpful for us that cannot upgrade to 0.10.x yet. Kind regards, Stevo Slavic.

Re: Migrating old consumer offsets to new consumer

Hello Nick, There are migration instructions in the docs (see http://kafka.apache.org/documentation.html#offsetmigration ) Kind regards, Stevo Slavic. On Wed, Sep 21, 2016 at 5:32 PM, Nick Kleinschmidt wrote: > I’m running the old 0.8 consumer storing offsets in

Re: Balancing based on lag

Hello Jens, By design there can be only one active consumer per consumer group per partion at a time, only one thread after being assigned (acquiring a lock) moves the offset for consumer group for partition, so no concurrency problems. Typically that assignment lasts long until rebalancing gets

Re: kafka latency

Hello Chao, How did you measure latency? See also https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Kind regards, Stevo Slavic. On Tue, Jul 26, 2016 at 9:52 PM, Luo, Chao wrote: > Dear Kafka guys, > > I measured

Re: Kafka Replication Factor

Hello Avi, Yes, replication factor of 1 means just one lead replica gets assigned for the topic partitions. To achieve HA of messages there are more things to watch out, not only replication factor - metadata/ZK HA, producer acks, min-in-sync replicas, rack awareness for replica assignment to

ZkUtils creating parent path with different ACL

Hello Apache Kafka community, Is there a reason why acls are not passed through in https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L404 call ? If one overrides and uses custom acls for a node, parent if missing might get created with default ACL. Kind

Re: PartitionMetadata hasLeader

Correction, leader was an Option, and I guess Optional cannot be used yet because Java language level compatibility is too low. What a mess, Java and Scala in codebase, and even new code being written in old Java. On Mon, Jun 20, 2016 at 12:08 PM, Stevo Slavić <ssla...@gmail.com> wrote: &

PartitionMetadata hasLeader

Hello Apache Kafka community, Scala API had hasLeader method in PartitionMetadata of MetadatResponse. I see new Java API does not have such method and there's no javadoc on the class. Can somebody please explain - will leader reference be null or something when partition has no leader? Why not

Re: [DISCUSS] Java 8 as a minimum requirement

+1 on Java 8 already, Scala 2.11 is enough as well On Fri, Jun 17, 2016 at 9:48 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > +1 to Java8 - we have already moved to Java8 last year. > > On Fri, Jun 17, 2016 at 12:48 PM, Ismael Juma wrote: > > > Hi Harsha, >

Re: Three consumers on a single partition

Replication factor should not be a problem, only lead is used for writes and reads. I'd monitor consumer offsets of all consumers of that topic - to make sure groupid is really same. Another thing could be that sarama client is simple, not implementing high level consumer like locking that only

Kafka controller replica state docs outdated?

Hello Apache Kafka community, Is it intentional that not all states (like ReplicaDeletionIneligible) are documented on https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals or is the page just outdated? Btw I see replica states documented in javadoc

UnderReplicatedPartitions 0 while lead is only ISR member

Hello Apache Kafka community, On 0.9.0.1 cluster, with all brokers up, I have a topic with single partition and replication factor of 3, min ISR is 2. When topic was created all 3 assigned replicas are in ISR. Now: 1. All brokers report UnderReplicatedPartitions of 0 while describe topic reports

Re: New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Thanks for sharing Ismael! On Fri, Apr 1, 2016 at 6:24 PM, Ismael Juma <ism...@juma.me.uk> wrote: > See: > > https://github.com/apache/kafka/pull/645 > https://github.com/apache/kafka/pull/735 > > Ismael > > On Fri, Apr 1, 2016 at 5:04 PM, Stevo Slavić <ssla...@g

Re: New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

, 2016 at 5:52 PM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello Apache Kafka community, > > With kafka-clients 0.9.0.1, async producer doesn't seem to handle deleted > topics well. > > If I create a topic, publish few messages to it, and delete it, > KafkaProdu

New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Hello Apache Kafka community, With kafka-clients 0.9.0.1, async producer doesn't seem to handle deleted topics well. If I create a topic, publish few messages to it, and delete it, KafkaProducer instance and underlying NetworkClient which was used to publish to that topic, will endlessly every

Re: How to get message count per topic?

See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Using metadata api one can get topic partitions and for each partition which broker is lead. Using offset api one can get partition size. Both apis are low level and to use them directly you would use

Re: Deletion of topic on 0.9.0.0 spams this exception

I've recently created related ticket https://issues.apache.org/jira/browse/KAFKA-3390 On Mon, Mar 14, 2016, 20:54 Scott Reynolds wrote: > >Conditional update of path > >/brokers/topics/messages.events/partitions/0/state with data > >

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

You are affected by this 0.9.0.0 bug https://issues.apache.org/jira/browse/KAFKA-2988 It was fixed for 0.9.0.1. You could just apply same fix to your 0.9.0.0 cluster but I'd recommend upgrading to 0.9.0.1. Kind regards, Stevo Slavic. On Mon, Mar 14, 2016, 07:10 Kunal Gupta

Re: Kafka topic deletion still not working well

the state node. Will check more logs, Kafka and ZK, if I can find what deleted the state node. Any ideas? On Fri, Mar 11, 2016 at 5:22 PM, Stevo Slavić <ssla...@gmail.com> wrote: > Sorry, I do not know if this is related to KAFKA-2937, I let you judge and > decide whether ticket needs to

Re: Kafka topic deletion still not working well

was updated by > a > > controller and then the controller dies and the new controller gets in to > > the exception and never completes deleting the topic. The topic existed > in > > the cluster and was also marked for delete. > > The case reported here is that the topic d

Re: Kafka topic deletion still not working well

etionSuccessful) (kafka.controller.ReplicaStateMachine) Kind regards, Stevo Slavic. On Sun, Mar 6, 2016 at 12:31 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Thanks Stevo, > > Feel free to paste your findings in KAFKA-2937, we can re-open that ticket > if necessary. > > Guozhang > >

test-log4j.properties purpose

Hello Apache Kafka community, Is config/test-log4j.properties used for tests only? If so, would it make sense to remove it from binary distribution archive? It's in kafka_2.11-0.9.0.1.tgz Kind regards, Stevo Slavic.

Kafka topic deletion still not working well

Hell Apache Kafka community, I'm still investigating an incident; from initial findings topic deletion doesn't seem to work well still with Kafka 0.9.0.1, likely some edge case not covered. Before with 0.8.2.x it used to happen that non-lead replica would be stuck in topic deletion process, and

Re: Apache Kafka Case Studies

Hello Joe, Consider registering for http://www.oreilly.com/online-training/building-distributed-pipelines-for-data-science.html and/or for http://www.kafka-summit.org/ Just as an idea where you could get ideas :) Btw, there's no such thing as best practice - only practice/approach that worked in

Re: Group coordinator in 0.9

Hello Oleksiy, All offsets for given consumer group are stored in (and fetched from) one and only one partition of all partitions of __consumer_offsets topic. By default __consumer_offsets has 50 partitions and for each of these partitions replication factor is 3, so there is one lead and 2 extra

Default max retries diff between old and new producer

Hello Apache Kafka community, Old producer had message.send.max.retries defaulting to 3, while new producer retries config property defaults to 0. Can somebody please explain what is the rationale to use 0 retries as default in new producer? Kind regards, Stevo Slavic.

min ISR of 1 for consumer offsets topic and disabling unclean leader election - problem or not?

Hello Apache Kafka community, Consumer offsets topic default configuration is somewhat different from all other topics for a good reason, e.g. replication factor is 3 compared to 1 for other topics, and number of partitions is 50 compared to 1 for other topic. Can somebody please explain why does

Re: Migrate a topic which has no brokers

Hello Ben, Yes, you can use apply different replica assignment. See related docs: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Kind regards, Stevo Slavic. On Tue, Jan 5, 2016 at 5:32 PM, Ben Davison wrote: > Hi All, > > On our dev

Re: Does MirrorMaker delete topics?

AFAIK "if the right options are set" actually means "if auto.create.topics.enable is left to default, set to true". As on any Kafka cluster with this configuration option, this will allow implicit topic creation, e.g. on first message being published to the topic, if topic does not exist it will

Re: Does MirrorMaker delete topics?

uster replicate topics onto the secondary site but don't take part > in topic leadership - is this on the cards at all? > > Thanks, > SimonC > > -----Original Message- > From: Stevo Slavić [mailto:ssla...@gmail.com] > Sent: 05 January 2016 14:04 > To: users@kafka.ap

Re: How to reset a consumer-group's offset in kafka 0.9?

Have you considered deleting and recreating topic used in test? Once topic is clean, read/poll once - any committed offset should be outside of the range, and consumer should reset offset. On Tue, Dec 29, 2015 at 4:11 PM, Han JU wrote: > Hello, > > For local test purpose

Re: How to reset a consumer-group's offset in kafka 0.9?

n starts from the beginning and consumers all the messages. > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ssla...@gmail.com>: > > > Have you considered deleting and recreating topic used in test? > > Once topic is clean, read/poll once - any committed offset should be >

Re: Unable to set log.cleanup.policy on topic

Hello Rakesh, log.cleanup.policy is broker configuration property, while cleanup.policy is topic configuration property (see http://kafka.apache.org/documentation.html#topic-config ). Since you are configuring particular topic, you need to use second one. Kind regards, Stevo Slavic. On Mon, Dec

Re: Failed attempt to delete topic

Delete was actually considered to be working since Kafka 0.8.2 (although there are still not easily reproducible edge cases when it doesn't work well even in in 0.8.2 or newer). In 0.8.1 one could request topic to be deleted (request gets stored as entry in ZooKeeper), because of presence of the

Re: Problems getting offsets

he kafka/zookeeper system? Will it affect how logs are managed, > and when “older” messages are purged? Or are they two independent systems? > > On 11/2/15, 03:51, "Stevo Slavić" <ssla...@gmail.com> wrote: > > >Here is a bit longer and more detailed, not necessari

Blocked consumer - next offset same as current offset

Hello Apache Kafka community, I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some conditions fetch response message set for a partition can contain at least one (if not all) MessageAndOffset with nextOffset being equal to current (committed) offset, offset used in fetch

Re: Blocked consumer - next offset same as current offset

would get triggered only when consumer lag is of appropriate size - I guess async producer made this issue more likely, because of batch sending multiple messages to same partition. On Wed, Nov 11, 2015 at 9:40 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello Apache Kafka community,

Re: Blocked consumer - next offset same as current offset

me compressed batch. It's > the responsibility of the client to skip over those messages. Note that the > high level consumer handles that logic already. > > Thanks, > > Jun > > On Wed, Nov 11, 2015 at 12:40 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Hel

Scheduled job in Kafka, every 2h?

Hello Apache Kafka community, I'm seeing some cyclic load on cluster, trying to determine cause. Couldn't determine from docs - is there some internal scheduled job in Kafka (0.8.2.1) executing on every broker every two hours? Kind regards, Stevo Slavic.

Re: Problems getting offsets

opic, but with internal consumer offsets topic. On Mon, Nov 2, 2015 at 1:56 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello David, > > In short, problem is not with your topic, it is with consumer offsets > topic initialization. > > You could modify your code to just r

New producer and storing offsets in Kafka - previously committed offsets fetched as uncommitted

Hello Apache Kafka community, I'm trying to use new producer, from kafka-clients 0.8.2.2, together with simple consumer to fetch and commit offsets stored in Kafka, and I'm seeing strange behavior - a committed offset/message gets read multiple times, offset fetch requests do not always see

Re: ("offsets.storage","kafka") not working with Consumer API

Hello Kiran, Check how many brokers you have in the cluster. Consumer offsets topic requires by default at least 3. In dev environment you could lower replication factor for that topic (see broker config options). Kind regards, Stevo Slavic. On Fri, Oct 16, 2015, 07:31 Kiran Singh

Re: How to verify offsets topic exists?

, Stevo Slavić <ssla...@gmail.com> wrote: > There's another related bug - triggering offsets topic creation through > requesting metadata about that topic does not work in case of single broker > clean (no topics created yet) Kafka cluster running. In that case sequ

Re: How to verify offsets topic exists?

factor for offsets topic. Does this make sense? As workaround, I guess I will have to resort to explicitly creating offsets topic if it doesn't exist already. Kind regards, Stevo Slavic. On Tue, Oct 6, 2015 at 11:34 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Debu

Re: How to verify offsets topic exists?

ls /brokers/topics/__consumer_offsets > > get /brokers/topics/__consumer_offsets > > > Thanks, > Grant > > On Mon, Oct 5, 2015 at 10:44 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > Hello Apache Kafka community, > > > > In my integration tests,

Re: How to verify offsets topic exists?

vic. On Tue, Oct 6, 2015 at 10:02 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Thanks Grant for quick reply! > > I've used AdminUtils.topicExists("__consumer_offsets") check and even > 10sec after Kafka broker startup, the check fails. > > When, on which event,

How to verify offsets topic exists?

Hello Apache Kafka community, In my integration tests, with single 0.8.2.2 broker, for newly created topic with single partition, after determining through topic metadata request that partition has lead broker assigned, when I try to reset offset for given consumer group, I first try to discover

Re: New Consumer & committed offsets

Hello Damian, Yes, there's a +1 difference. See related discussion http://mail-archives.apache.org/mod_mbox/kafka-users/201507.mbox/%3CCAOeJiJh2SMzVn23JsoWiNk3sfsw82Jr_-kRLcNRd-oZ7pR1yWg%40mail.gmail.com%3E Kind regards, Stevo Slavic. On Tue, Sep 15, 2015 at 3:56 PM, Damian Guy

Re: Unclean leader election docs outdated

Created https://issues.apache.org/jira/browse/KAFKA-2551 On Mon, Sep 14, 2015 at 7:22 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Yes you are right. Could you file a JIRA to edit the documents? > > Guozhang > > On Fri, Sep 11, 2015 at 4:41 PM, Stevo Slavić <

Re: open source projects based on kafka

Have a look at https://github.com/allegro/hermes On Mon, Sep 14, 2015, 01:28 David Luu wrote: > The toy project idea is good. Another option I think could be to look at > the various Kafka client langague bindings and/or utilities (like > kafkacat). And from there, another

Re: Unclean leader election docs outdated

t;, mentions ability to disable unclean leader election, so likely just this one reference needs to be updated. On Sat, Sep 12, 2015 at 1:05 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Stevo, > > Could you point me to the link of the docs? > > Guozhang > > On F

Unclean leader election docs outdated

Hello Apache Kafka community, Current unclean leader election docs state: "In the future, we would like to make this configurable to better support use cases where downtime is preferable to inconsistency. " If I'm not mistaken, since 0.8.2, unclean leader election strategy (whether to allow it

Re: [DISCUSSION] Kafka 0.8.2.2 release?

and also including https://issues.apache.org/jira/browse/KAFKA-2114 Thanks Jeff On Sun, Aug 16, 2015 at 2:51 PM, Stevo Slavić ssla...@gmail.com wrote: +1 (non-binding) for 0.8.2.2 release Would be nice to include in that release new producer resiliency bug

Re: [DISCUSSION] Kafka 0.8.2.2 release?

+1 (non-binding) for 0.8.2.2 release Would be nice to include in that release new producer resiliency bug fixes https://issues.apache.org/jira/browse/KAFKA-1788 and https://issues.apache.org/jira/browse/KAFKA-2120 On Fri, Aug 14, 2015 at 4:03 PM, Gwen Shapira g...@confluent.io wrote: Will be

Re: 0.8.3 ETA?

://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan). As we are getting closer, we will clean up the 0.8.3 jiras and push non-critical ones to future releases. Thanks, Jun On Mon, Aug 3, 2015 at 5:52 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community

0.8.3 ETA?

Hello Apache Kafka community, If I recall well, two weeks ago it was mentioned in a discussion that Kafka 0.8.3 might be released in a month time. Is this still Kafka dev team goal, in few weeks time to have Kafka 0.8.3 released? Or is more (re)work (e.g. more new consumer API changes) planned

Re: 0.8.2 Producer sends blocking when brokers are unavailable

Hello David, It's a known issue, see https://issues.apache.org/jira/browse/KAFKA-1788 and https://issues.apache.org/jira/browse/KAFKA-2120 Kind regards, Stevo Slavic. On Fri, Jul 31, 2015 at 10:15 AM, David KOCH dk...@ezakus.com wrote: Hello, The new producer

Re: New consumer - offset one gets in poll is not offset one is supposed to commit

, Jason On Fri, Jul 24, 2015 at 10:11 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, Say there is only one topic with single partition and a single message on it. Result of calling a poll with new consumer will return ConsumerRecord for that message

Re: New consumer - partitions auto assigned only on poll

to keep it non-blocking. Perhaps you can open a ticket and we can get feedback from some other devs? Thanks, Jason On Wed, Jul 22, 2015 at 2:09 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In the new consumer I encountered unexpected behavior. After constructing

New consumer - offset one gets in poll is not offset one is supposed to commit

Hello Apache Kafka community, Say there is only one topic with single partition and a single message on it. Result of calling a poll with new consumer will return ConsumerRecord for that message and it will have offset of 0. After processing message, current KafkaConsumer implementation expects

Re: New consumer - poll/seek javadoc confusing, need clarification

election is not enabled). On Tue, Jul 21, 2015 at 9:11 PM, James Cheng jch...@tivo.com wrote: On Jul 21, 2015, at 9:15 AM, Ewen Cheslack-Postava e...@confluent.io wrote: On Tue, Jul 21, 2015 at 2:38 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, I

Re: New consumer - poll/seek javadoc confusing, need clarification

Strange, if after seek I make several poll requests, eventually it will read/return messages from offset that seek set. On Thu, Jul 23, 2015 at 11:03 AM, Stevo Slavić ssla...@gmail.com wrote: Thanks Ewen for heads up. It's great that seek is not needed in between poll when business goes

Re: New consumer - ConsumerRecords partitions

the partitions which had messages. Would you mind creating a jira for the feature request? You're welcome to submit a patch as well. -Jason On Tue, Jul 21, 2015 at 2:27 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, New HLC poll returns ConsumerRecords. Do

Re: High Level Consumer use case

Hello Sreenivasulu, It's automatic, just start them, and as each HLC starts it registers in ZK, rebalancing of the HLC to partition assignments happens. Be gentle when starting consumers, there is a bug reported that if multiple HLCs are started in short time, some of them may end up without any

Re: New consumer - consumer group init

, Jul 20, 2015 at 11:06 AM, Jason Gustafson ja...@confluent.io wrote: Hey Stevo, The new consumer doesn't have any threads of its own, so I think construction should be fairly cheap. -Jason On Sun, Jul 19, 2015 at 2:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Guozhang

New consumer - ConsumerRecords partitions

Hello Apache Kafka community, New HLC poll returns ConsumerRecords. Do ConsumerRecords contain records for every partition that HLC is actively subscribed on for every poll request, or does it contain only records for partitions which had messages and which were retrieved in poll request? If

New consumer - poll/seek javadoc confusing, need clarification

Hello Apache Kafka community, I find new consumer poll/seek javadoc a bit confusing. Just by reading docs I'm not sure what the outcome will be, what is expected in following scenario: - kafkaConsumer is instantiated with auto-commit off - kafkaConsumer.subscribe(someTopic) -

New producer hangs inifitely when it looses connection to Kafka cluster

Hello Apache Kafka community, Just noticed that : - message is successfully published using new 0.8.2.1 producer - and then Kafka is stopped next attempt to publish message using same instance of new producer hangs forever, and following stacktrace gets logged repeatedly: [WARN ]

Re: New consumer - consumer group init

...@gmail.com wrote: Hi Stevo, Hmm this is interesting, do you have any use cases in mind that need dynamic group changing? Guozhang On Fri, Jul 17, 2015 at 11:13 PM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only

New consumer - consumer group init

Hello Apache Kafka community, In new KafkaConsumer API on trunk, it seems it's only possible to define consumer group id at construction time of KafkaConsumer, through property with group.id key. Would it make sense and be possible to support setting/changing consumer group id after

Re: Delete topic using Admintools is not working

Hello Sivananda, Calling AdminUtils.deleteTopic just requests topic to be deleted - it does not actually delete topic immediately. Requests for topic deletion get saved in ZooKeeper as a node (named by topic name), under /admin/delete_topics node. If brokers in the cluster are configured with

Custom topic metadata

Hello Apache Kafka Community, Is it possible to store and retrieve additional custom topic metadata along with existing Kafka managed ones, using some Kafka API? If not would it be a problem (e.g. for Kafka broker or some client APIs) if I was to store/retrieve additional custom topic metadata

Re: kafka connection with zookeeper

Hello Gaurav, There are several Zookeeper client connection configuration properties you can tune for Kafka brokers. They are document @ http://kafka.apache.org/documentation.html#brokerconfigs (all begin with zookeeper. prefix) Did you change any already? I'd start with defaults (they can and

Re: Kafka settings for (more) reliable/durable messaging

. A replica can't reach offset of second batch, without first having written the first batch. So I believe we are safe in this scenario. Gwen On Tue, Jul 7, 2015 at 8:01 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Gwen, Thanks for fast response! Btw, congrats on officially becoming

Kafka settings for (more) reliable/durable messaging

Hello Apache Kafka community, Documentation for min.insync.replicas in http://kafka.apache.org/documentation.html#brokerconfigs states: When used together, min.insync.replicas and request.required.acks allow you to enforce greater durability guarantees. A typical scenario would be to create a

Re: Kafka settings for (more) reliable/durable messaging

ka-49753844 Thanks, Jiangjie (Becket) Qin On 7/7/15, 9:28 AM, Stevo Slavić ssla...@gmail.com wrote: Thanks for heads up and code reference! Traced back required offset to https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/serve r/ReplicaManager.scala#L303 Have

Re: Kafka settings for (more) reliable/durable messaging

different replica can ACK the second back of messages while not having the first - from what I can see, it will need to be up-to-date on the latest messages (i.e. correct HWM) in order to ACK. On Tue, Jul 7, 2015 at 7:13 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community

Re: failing to fetch topic metadata

Hello Emanuele, From logs it seems that auto.create.topics.enable is not overriden for the embedded broker. It also seems that test is explicitly creating topic before publishing message to it. Consider commenting out explicit topic creation and rely on implicit topic creation. Kind regards,

Re: Indication in FetchResponse that fetch size is too small

that your fetch size needs to increase. Thanks, Joel On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote: Hello Apache Kafka community, Couldn't broker return a special error code in FetchResponse for a given partition(s) where it detects that there was something to return

  1   2   >