SocketTimeoutException with kafka-producer-perf-test.sh

Hello Folks, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have cre

Re: Possible DEAD LOCK for one day at broker controller?

Hi Zhao, Do you see any other errors regarding checkpoint file? Is this reproducible by you and if you can you enable debug log level to get more info.​ On Thu, Aug 20, 2015 at 7:44 AM, Zhao Weinan wrote: > Hi Kishore Senji, > > I've been busy recovering some data these two days... and found tha

is SSL support feature ready to use in kafka-truck branch

Hi folks, I tried to clone the latest version of kafka truck and try to enable the SSL. The server.properties seems not having any security related settings, and it seems there's no other config file relevant to SSL either. So may I know is this feature ready to use now in truck branch? BTW, we're

Re: Possible Memory Leak in Record Accumulator

The producer did not recover on its own. The app froze for upwards of 10 minutes. Long after net connectivity was restored. I could telnet to port 9092 on the broker in the error logs. After collecting the heap dump I had to kill and restart the app. The stack trace of the producer-network-threa

Re: Possible Memory Leak in Record Accumulator

It will also be interesting to see stack trace from "kafka-producer-network-thread" (which is the one that should be sending the batches but maybe got stuck), and if this issue is reproducible for you in a test environment - maybe generate logs in TRACE level. On Thu, Aug 20, 2015 at 5:35 PM, Gwen

Re: Possible Memory Leak in Record Accumulator

Hi, I didn't see this issue during our network hiccups. You wrote you saw: Got error produce response with correlation id 17717 on topic-partition event.beacon-38, retrying (8 attempts left). Error: NETWORK_EXCEPTION What did you see after? Especially once the network issue was resolved? more re

Possible Memory Leak in Record Accumulator

I've been running into an issue with the 0.8.2.1 new producer for a few weeks now and I haven't been able to figure it out. Hopefully someone on the list can help! First off my producer config looks like this: props.put(ProducerConfig.ACKS_CONFIG, "1") props.put(ProducerConfig.RETRIES_CO

Any drawbacks of running Kafka consumer in a web container

Hello guys, Does anyone run Kafka consumer in a web container like Tomcat in their production environments (processing millions of messages per hour). I am wondering if the web container could have any adverse impact on the performance of consumer. Is it advisable to NOT run the consumer in a we

Re: thread that handle client request in Kafka brokers

Thanks Ben. I think what I heard is that in broker socketServer it only has one thread(acceptor?) to handle new connections which may potentially become a bottleneck if many small requests come in. -Tao On Mon, Aug 17, 2015 at 7:19 AM, Ben Stopford wrote: > Hi Tao > > This is unlikely to be a pr

Leader and Replicas order out of sync.

Our 2 kafka brokers ( 1 & 5) were rebooted due to hypervisor going down and I think we encountered a similar issue that was discussed in thread "Problem with node after restart no partitions?". The resulting JIRA is closed without conclusions

Re: Write-time based synchronization of message consumption from multiple topics

Hello Petr, Thank you for your reply. I am not sure how your proposed solution could help my use case. Sending messages with the same key to the same partition will ensure that key : (message_type, value) that share a key will be consumed by the same consumer instance. However, my main concern wa

Re: Problem with node after restart no partitions?

Hi, I've been through some similar problems, posted in anthor thread (I just pasted some broker logs, please check the latest message which the mail archieve

Re: Possible DEAD LOCK for one day at broker controller?

Hi Kishore Senji, I've been busy recovering some data these two days... and found that I maybe hit more serious problem than I thought. I lost almost all data on one broker at least at some time, here is some log from server.log pasted below, and very like the situation described by Jason and Thun

Re: How to skip malformed messages in a custom Kafka decoder?

I'm using Spark DirectKafkaInputDStream so I don't have direct access to the consumer. There is no option to push down message filter so that decoded messages are dropped before InputDStream is created. Hence I have to decode everything into RDD then transform to a new RDD. I was thinking about my

Re: Write-time based synchronization of message consumption from multiple topics

Hi, to my knowledge this is generally solved by partitioner which logs messages with the same key to the same topic. It is not clear why you can't use this approach. With Regards, Petr On Thu, Aug 20, 2015 at 12:35 PM, David KOCH wrote: > Hello, > > Is it possible to consume messages from multi

Re: Patch for SSL setup for Kafka

Hi LCassa, The SSL/TLS patch was recently merged to trunk, so it should be easier to test. Best, Ismael On Wed, Aug 19, 2015 at 12:54 AM, Cassa L wrote: > Thank You! I will look into it. > > On Tue, Aug 18, 2015 at 10:34 AM, Sriharsha Chintalapani > wrote: > > > Hi , > > you can use the l

Write-time based synchronization of message consumption from multiple topics

Hello, Is it possible to consume messages from multiple topics in a way that messages which were written at the same time in different topics are also consumed at around the same time? Let me explain this with our use case. Our application logs keyed messages of N different types into N topics:

Exception in logs, on latest Kafka Java client - 0.8.2.1

>2015-08-20 16:55:43,242 [kafka-producer-network-thread | producer-1] DEBUG >o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node >0 >2015-08-20 16:55:43,246 [kafka-producer-network-thread | producer-1] DEBUG >o.apache.kafka.clients.NetworkClient - Init connection to nod