CSharp librari and Producer Closing socket for because of error (kafka.network.Processor),java.nio.BufferUnderflowException

Hi I have kafka broker running (kafka_2.9.1-0.8.1.1) All is working. One project requires producer is written in CSharp I am not dot net programmer but I managed to write simple producer code using https://github.com/kafka-dev/kafka/blob/master/clients/csharp/README.md the code ... using

Re: Kafka producer in CSharp

Thank you for response. I think HTTP is ok. I have two more question in case of HTTP. 1. Can I have fault tolerance in case I have two or more brokers? 2. Can I ack that message is in queue? Best regards, Margus (Margusja) Roo +372 51 48 780 http://margus.roo.ee

Re: Kafka producer in CSharp

Ok got some info myself. I can have fault tolerance - I can start kafka-http-endpoint using broker lists I can have ack - start using --sync But what is best practice in case if kafka-http-endpoint goes down? Start multiple kafka-http-endpoint's and in client side just control that

TCP_TOO_MANY_ESTAB_CONNS

Dear all, We started using kafka as a queue server recently. Our production setup has 3 kafka servers and 3 zookeepers. We having 30 topics with 10 partitions each. We have producer pool(10 producers / APP_SERVERS). So totally 30 producers. With our monitoring tool, we could detect

What happens to Kafka when ZK lost its quorum?

Hi all, Can Kafka producers, brokers and consumers still be processing messages and functioning in their normal states if Zookeeper lost its quorum? Thanks, Connie

Re: Kafka producer in CSharp

The wire protocol has changed drastically since then. I don't know of any C# clients (there are none on the client library page nor have I heard of any being used in production but maybe there are some). For clients that use DotNet I often suggest that they use some HTTP producer/consumer

Re: What happens to Kafka when ZK lost its quorum or becomes unstable?

The brokers are designed with the expectation that ZK service is always available. If ZK is not available, the clients may not be able to send/fetch data properly. Thanks, Jun On Tue, May 13, 2014 at 6:48 AM, Connie Yang cybercon...@gmail.com wrote: Hi, Can the producers, brokers and

Re: Loss of Leader in Kafka

3. Deleted the topic. Checked only the zookeeper to see if the /brokers/topics DOES NOT have the topic You are seeing this problem since delete topic is not supported in Kafka. Any attempt to delete a topic may leave your cluster in an unstable state. We plan to release 0.8.2 with delete topic

What happens to Kafka when ZK lost its quorum or becomes unstable?

Hi, Can the producers, brokers and consumers still be processing messages when their ZK cluster lost its quorum or becomes unstable? I know this is rather general question as it may depends on what configuration these use. So, please enumerate all of those combinations. Thanks, Connie

100 MB messages

Hello everyone, Can Kafka be used for binary large objects of 100 MB ? Or should I use a different solution to store these files like MongoDB and maybe send the location of these files in MongoDB over Kafka? Thanks is advance, Wouter

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

In 0.8, ZK is critical for managing the brokers. So, we do expect that ZK service to be always available. Thanks, Jun On Mon, May 12, 2014 at 9:59 AM, Yury Ruchin yuri.ruc...@gmail.com wrote: Hi all, I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. Steps to

Re: TCP_TOO_MANY_ESTAB_CONNS

Hi Ranjith, Which version of Kafka are you using? Guozhang On Tue, May 13, 2014 at 3:53 AM, Ranjith Venkatesan ranjit...@zohocorp.comwrote: Dear all, We started using kafka as a queue server recently. Our production setup has 3 kafka servers and 3 zookeepers. We having 30 topics

Re: 100 MB messages

Or HDFS and use Kafka for the event of file, yup. Processing on the files can be done without the mapreduce overhead in Hadoop now using Apache Tez (or something that use Tez like Pig). /*** Joe Stein Founder, Principal Consultant Big Data Open

Killing last replica for partition doesn't change ISR/Leadership if replica is running controller

Hi, Kafka version is 0.8.1.1. We have three machines: A,B,C. Let’s say there is a topic with replication 2 and one of it’s partitions - partition 1 is placed on brokers A and B. If the broker A is already down than for the partition 1 we have: Leader: B, ISR: [B]. If the current controller is

Re: Loss of Leader in Kafka

Hi Kashyap, This may be a real issue with delete-topic feature, could you also reproduce this with trunk HEAD and 1 partition/1 replica? Guozhang On Mon, May 12, 2014 at 9:39 AM, Kashyap Mhaisekar kashya...@gmail.comwrote: Hi, I am hitting a strange exception while creating a topic in Kafka

Re: Kafka 0.7.X maven repo

Maven wasn't introduced until 0.8.0. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /

zookeeper down

Can kafka survive when zookeeper is down and not connectable ? Will the consumer or producer still work in that case ? Weide

0.7 maven repo

We are currently using kafka 0.8.X but want to try something with 0.7.x and compare. Can anyone point me to the kafka 0.7.X maven repo ? Thanks in advance Vijay

Re: 100 MB messages

It can, but it will not perform very well. Kafka fully instantiates messages in memory (as a byte[] basically) so if you send a 100MB message the server will do a 100MB allocation to hold that data prior to writing to disk. I think MongoDB does have blob support so passing a pointer via Kafka as

kafka performance question

Dear all, We want to use kafka to collect and dispatch data file, but the performance is maybe lower than we want. In our cluster,there is a provider and a broker. We use a one thread read file from local disk of provider and send it to broker. The average throughput is only 3 MB/S~4MB/S. But

Re: Loss of Leader in Kafka

After topic deletion, I deleted the physical folders in data logs and then restarted Kafka. That resolved. Thanks! Will try out 0.8.1 and let the group know. Regards, Kashyap On Monday, May 12, 2014, Jun Rao jun...@gmail.com wrote: Delete topic doesn't quite work in 0.8.1. We recently fixed it

Re: keyed-messages de-duplication

Hi, The compaction is done to clean-up space. It isn't done immediately only periodically. I suspect the reason you see no compaction is that we never compact the active segment of the log (the most recent file) as that is still being written to. The compaction would not happen until a new

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

Hi Yury, When auto-commit is turned on, before the consumer shuts down completely it needs to commit its last offset to ZK to avoid any duplicates consumed when it starts over. If the ZK is not up running then the shutdown would rather stuck than let it go and incur duplicates. This behavior is

Re: Killing last replica for partition doesn't change ISR/Leadership if replica is running controller

Yes, that seems like a real issue. Could you file a jira? Thanks, Jun On Tue, May 13, 2014 at 11:58 AM, Alex Demidko alexan...@metamarkets.comwrote: Hi, Kafka version is 0.8.1.1. We have three machines: A,B,C. Let’s say there is a topic with replication 2 and one of it’s partitions -

Kafka 0.7.X maven repo

We are currently using kafka 0.8.X but want to try something with 0.7.x and compare. Can anyone point me to the kafka 0.7.X maven repo ? Thanks in advance Vijay