Re: unable to delete a topic

What I ended up doing, after having similar issues your having, was: - stop all the brokers - rm -rf all the topic data across the brokers - delete the topic node in ZK - set auto.create.topics.enable=false in the server.properties - start the brokers up again The topic stayed deleted this

Re: Apache Kafka Case Studies

Hello Joe, Consider registering for http://www.oreilly.com/online-training/building-distributed-pipelines-for-data-science.html and/or for http://www.kafka-summit.org/ Just as an idea where you could get ideas :) Btw, there's no such thing as best practice - only practice/approach that worked in

Merging Topics

Hello Kafka, I would like to know how I can connect two kafka servers together and merge a topic with the same name. And a safer way to detect whether a topic is deleted completely from the cluster when adding a new complete server to the cluster (this node has a my custom application which

Kafka-streams restoreActiveState

I am trying to use kafka-streams (I know it isn’t released yet but seems to be fairly far on in its implementation and I would like to use that rather than samza for this project as my project doesn’t need to go live yet). My set up has a single source, producer, in memory state store and sink.

Announcing ruby-kafka v0.1

I've implemented a new Kafka client in Ruby: https://github.com/zendesk/ruby-kafka Currently I'm focusing on providing a rock solid Producer API (since that was the use case prompting me to write my own library) including graceful failure handling, improved logging and instrumentation. A Kafka

Apache Kafka Case Studies

Dear Kafka users, I'm looking for some case studies around using Kafka on big projects. Specifically, I'm looking for some architectural insights into how I could orchestrate my data pipeline using Kafka on an enterprise system. Some pointers on some architectural best practices, slides on how

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Becket, On Wed, Jan 27, 2016 at 10:51 PM, Becket Qin wrote: > 2. For seek(), pause(), resume(), it depends on how easily user can use > them. > If we take current interface, and user have a list of partitions to > pause(), what they can do is something like: >

Number of concurrent consumers per data node

Hi I’m just coming up to speed with Kafka. Some beginner questions, may be point me to where I can find the answers please: 1. In a Kafka cluster what determines the maximum number of concurrent consumers that may be connected to a data node? 2. What is a/the scale out strategy for a Kafka

Re: Kafka-streams restoreActiveState

Thanks for letting me know. From what you said it seems a bug in the Kafka Streams, would you like to create JIRA with your topology code? I'll start the investigation right away. Guozhang On Wed, Feb 3, 2016 at 3:45 PM, Tom Dearman wrote: > I built trunk and pulled

Re: Kafka-streams restoreActiveState

I built trunk and pulled source yesterday. There are two topics created one called [store-name] the other as you describe [Job-Id]-[Store-name]-"changelog". Only the one called [store-name] contained the store state, the other topic contained nothing so to get this to work I had to compile the

Protocol Question

Hi, I am trying to navigate through the protocol and I am seeing some inconsistencies with the data. I am trying to parse out the MetadataResponse and I am seeing bytes in between where they shouldn't be. I know they are extra, because if I up the offset the data after is correct. Here is

Re: Apache Kafka Case Studies

Hi Joe, This might be interesting: https://engineering.linkedin.com/kafka/running-kafka-scale Cheers, Jens On Wed, Feb 3, 2016 at 4:15 PM, Joe San wrote: > Dear Kafka users, > > I'm looking for some case studies around using Kafka on big projects. > Specifically, I'm

Re: kafka 0.8.2.2 delete topic issue

I just ran into this issue in our load environment, unfortunately I came up with the same options outlined above. Any better solutions would be most appreciated otherwise I'm now considering the use of delete topic in any critical environment off the table. On Wed, Feb 3, 2016 at 10:10 AM Ivan

kafka 0.8.2.2 delete topic issue

Hi! We're running cluster of 3 kafka-0.8.2.2 nodes and delete.topic.enable is set to true on all nodes. Today we tried to delete one of the topics. I waited ~20 minutes after the kafka-topics.sh --delete was executed but the topic was still there. --describe showed Leader: -1 and only one of

Re: unable to delete a topic

Thanks. Setting auto.create.topics.enable=false made the difference. On Wednesday, February 3, 2016 2:45 PM, John Holland wrote: What I ended up doing, after having similar issues your having, was: - stop all the brokers - rm -rf all the topic data

Re: kafka “stops working” after a large message is enqueued

Deleted the topic and recreated (with max bytes set) but that did not help.What helped though is upping the java heap size.I monitored the consumer with jstat. I noticed 2 full garbage collection attempts right after publishing the large message. After that the consumer appeared dormant. Upping

Re: Producer code to a partition

In kafka, each record can have a key. This key is used to distribute records to partitions. All non-keyed records will be distributed in round-robin fashion. All keyed records will be distributed based on the hash of the key / or can write a custom partitioner. or we can specify partition number

Producer code to a partition

Kafka users, The code below is something that I have to write to a Topic! def publishMessage(tsDataPoints: Seq[DataPoint]): Future[Unit] = { Future { logger.info(s"Persisting ${tsDataPoints.length} data-points in Kafka topic ${producerConfig.topic}") val dataPoints =

Re: Producer code to a partition

Hi, You can use ProducerRecord(java.lang.String topic, java.lang.Integer partition, K key, V value) constructor to pass partition number. https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html Kumar On Thu, Feb 4, 2016 at 11:41 AM, Joe San

Re: Producer code to a partition

What is the partition key? Why do I need to specify the partition key and a partition number? On Thu, Feb 4, 2016 at 7:17 AM, Manikumar Reddy wrote: > Hi, > > You can use ProducerRecord(java.lang.String topic, java.lang.Integer > partition, K key, V value)

Re: Protocol Question

Hi Heath, a few comments: (1) you should be looping on brokerCount (2) you are missing the topic array count (4 bytes), and loop (3) topic error code is int16, so only 2 bytes not 4 (4) you are missing the partition metadata array count (4 bytes), and loop (5) you are missing the replicas and isr

Re: Protocol Question

Some comments based on your code snippet: (1) you aren't looping on brokerCount -- you should be decoding broker metadata for each count (2) you are missing the topic metadata array count (and loop) -- 4 bytes (3) topic errorcode is an Int16, so you should be reading 2 bytes, not 4 (4) you are

Re: Apache Kafka Case Studies

To follow up on the blog post of mine that Jens pasted, I also have a number of talk presentations up at http://www.slideshare.net/toddpalino If you download the actual presentation files, you’ll be able to access the slide notes in most of those that have a lot more information. On Wed, Feb 3,

Re: Apache Kafka Case Studies

The OReilly online training seems to be interesting! Is there anything else that is similiar to this that talks more of a real time experience from any organisation just like the one at LinkedIn? On Wed, Feb 3, 2016 at 6:10 PM, Todd Palino wrote: > To follow up on the blog

Re: Kafka-streams restoreActiveState

Hello Tom, Thanks for trying out Kafka Streams. A quick question: are you using Kafka Streams built from trunk? In the latest trunk we changed changelog topic name to be the store-name + "-changelog" suffix. So when your processing job starts the topology created state should have the topic name

unable to delete a topic

Try as I might I've found it impossible to delete a topic in kafka 0.9.0. I set delete.topic.enable in config.properties. I tried kafka-topics.sh with the delete command .I responds with: "topic marked for deletetion" but after a 30 minute wait the topic was till there. Then I stopped kafka and

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Most of the use cases of pause/resume that I've seen work only on single partitions (e.g in Kafka Streams), so the current varargs method is kind of nice. It would also be nice to be able to do the following: consumer.pause(consumer.assignment()); Both variants seem convenient in different

NullPointerException in calls of ZkClient

I am repeatedly seeing NPEs for calls made to ZkClient. I am using zkClient-0.5 + zookeeper 3.4.5 + kafka 0.8.2.0 There is a related jira for similar NPE issue, and looks like zkClient-0.5 must have solved the issue. https://issues.apache.org/jira/browse/KAFKA-2169

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

A good compromise would be to add an arity with a single TopicPartition. Jason Gustafson writes: > Most of the use cases of pause/resume that I've seen work only on single > partitions (e.g in Kafka Streams), so the current varargs method is kind of > nice. It would also be nice to be able to

Re: Apache Kafka Case Studies

Talk I gave about Kafka at the Wikimedia Foundation at Kafka NYC Meetup in 2014. https://www.hakkalabs.co/articles/apache-kafka-wikimedia On Wed, Feb 3, 2016 at 1:56 PM, Joe San wrote: > The OReilly online training seems to be interesting! Is there anything else > that