Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

2015-05-18 Thread Adam Dubiel
Hello everyone, I'm technical team lead of Hermes project. I will try to answer already posted questions, but feel free to ask me anything. 1) Can you comment on how this compares to Confluent's REST proxy? We do not perceive Hermes as mere proxy. While Confluent product wants to help services

Why is there no ZookeeperProducerConnector ?

2015-05-18 Thread Thomas Pocreau
Hi all, I'm looking for a simple way to create *a producer using zookeeper has source for metadata.broker.list.* I found the consumer equivalent : ZookeeperConsumerConnector but no sign of ZookeeperProducerConnector. Thanks for your help. -- *Thomas Pocreau* Recherche Développement |

Re: Partitions reassignment failed due to Partition reassignment currently in progress for Map()

2015-05-18 Thread Chris Neal
Perfect. Thank you very much! Chris On Sun, May 17, 2015 at 10:41 PM, Clark Haskins cl...@kafka.guru wrote: Yeah you can start using the reassignment tool again. It's likely some issue with the controller that put you in this state. Without gathering the logs it will be hard to tell.

Re: Experiences testing new producer performance across multiple threads/producer counts

2015-05-18 Thread tao xiao
Garry, Do you mind to share the source code that you did for the profiling? On Sun, May 17, 2015 at 4:59 PM, Garry Turkington g.turking...@improvedigital.com wrote: Hi Guozhang/Jay/Becket, Thanks for the responses. Regarding my point on performance dropping when the number of partitions

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

2015-05-18 Thread Warren Henning
Interesting. Thanks for sharing and working on this! Can you comment on how this compares to Confluent's REST proxy? On Sat, May 16, 2015 at 12:01 AM, Marcin Kuthan marcin.kut...@gmail.com wrote: Hi Everyone Hermes is an asynchronous message broker built on top of Kafka. It provides

Delete topic pending

2015-05-18 Thread Dillian Murphey
If a broker doesn't have the topic, and I run delete topic, that topic will be in a pending delete state forever. What am I doing wrong here? Also, what if I have data loss and I just want to delete the dang topic form zookeeper directly with non of this pending stuff. Thanks

Re: Delete topic pending

2015-05-18 Thread Sriharsha Chintalapani
May be thats a bug. We should probably alert the user saying the topic doesn’t exist rather than go and add it to the zookeeper for deletion trigger. Also to delete any topic you need to set “delete.topic.enable” to true --  Harsha On May 18, 2015 at 10:16:46 AM, Dillian Murphey

Re: Delete topic pending

2015-05-18 Thread Jiangjie Qin
Hmm, which Kafka version are you running? From the code it looks we should have already ignored the deletion of a topic if it does not exist. Jiangjie (Becket) Qin On 5/18/15, 10:15 AM, Dillian Murphey crackshotm...@gmail.com wrote: If a broker doesn't have the topic, and I run delete topic,

Which Kafka Consumer should I use?

2015-05-18 Thread luo.fucong
I want to rebuild my message queue from based on Redis to Kafka (because Redis persistence is terrible). Now I am using the new KafkaProducer, and I am wondering which consumer should I use. Currently there are 3 Kafka Consumers (until 0.8.2.1): 1. High Level Consumer 2. Simple Consumer 3.

kafkacat

2015-05-18 Thread clay teahouse
Hi All, Does anyone know of an implementation of kafkacat that reads from socket? Or for that matter any kafka producer client that can read from socket and publish to kafka? thanks Clay

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

2015-05-18 Thread Marcin Kuthan
Hi Stevo Thank's for your comment. On 16 May 2015 at 17:43, Stevo Slavić ssla...@gmail.com wrote: Nice, thanks for sharing! Is 30k msgs/sec publishing or push throughput? Will check, hopefully performance tests are included in sources. 30k msgs/sec is our regular traffic handled by hermes

Re: [Announcement] Hermes - pub / sub broker built on top of Kafka

2015-05-18 Thread Marcin Kuthan
Hi Warren With Hermes, the publisher is never blocked. Even if message has not been sent to Kafka cluster, or if message has been sent but not acknowledged. It is especially useful if your system needs to have strict SLA guarantees. From the consumers perspective there is retrying policy if the

SPARKTA: a real-time aggregation engine

2015-05-18 Thread David Morales
Hi there, We have released our real-time aggregation engine based on Spark Streaming. SPARKTA is fully open source (Apache2) and our goal is that no code would be required to work with it. The most interesting input is Apache Kafka and we are already working to improve the agreggation policies