Re: consumer.poll() takes approx. 30 seconds - 0.9 new consumer api

Rohit, The 30s number sounds very suspicious because it is exactly the value of the session timeout. But if you are driving the consumer correctly, you shouldn't normally hit this timeout. Dana was asking about consumers leaving gracefully because that is one case where you can inadvertently

Re: consumer.poll() takes approx. 30 seconds - 0.9 new consumer api

Can anybody help out on this? From: Rohit Sardesai Sent: 19 June 2016 11:47:01 To: users@kafka.apache.org Subject: Re: consumer.poll() takes approx. 30 seconds - 0.9 new consumer api In my tests , I am using around 24 consumer groups. I never call

Re: Fail fast producer/consumer when no connection to Kafka brokers cluster

You can adjust request.timeout.ms, which is shared between both new producer and new consumer. I don't think its quite what you want, but probably the closest that exists across both clients. There's not much more than that -- when you say "when the connection to the entire broker cluster is lost"

Re: [DISCUSS] Java 8 as a minimum requirement

Hi Ismael, Agree on timing is more important. If we give enough heads up to the users who are on Java 7 thats great but still shipping this in 0.10.x line is won't be good as it still perceived as maint release even the release might contain

Re: Kafka Connect HdfsSink and the Schema Registry

Great, glad you sorted it out. If the namespace is being omitted incorrectly from the request the connector is making, please file a bug report -- I can't think of a reason we'd omit that, but it's certainly possible it is a bug on our side. -Ewen On Wed, Jun 15, 2016 at 7:08 AM, Tauzell, Dave

Re: General Question About Kafka

The most common use case for Kafka is within a data center, but you can absolutely produce data across the WAN. You may need to adjust some settings (e.g. timeouts, max in flight requests per connection if you want high throughput) to account for operating over the WAN, but you can definitely do

Re: Consumer Question

Hi Chris, We should also ensure that auto.create.topics.enable is set to true. Thank you, Anirudh Hi Chris, If the topic not exist, it will create a new topic with the name which you give. Thanks, Nicole On Sat, Jun 18, 2016 at 1:55 AM, Chris Barlock wrote: > If you have

Re: Consumer Question

Hi Chris, If the topic not exist, it will create a new topic with the name which you give. Thanks, Nicole On Sat, Jun 18, 2016 at 1:55 AM, Chris Barlock wrote: > If you have a consumer listening on a topic and that topic is deleted is > the consumer made aware -- perhaps

Re: Wordcount with reduce

Hi Matthias, I solved the problem with specifying the serders and reading source as KStream instead of KTable. So, instead of KTable source = builder.table("topic1"); I added: KStream source = builder.stream(longSerde,stringSerde,"topic1"); Thanks

Re: Can I access Kafka Streams Key/Value store outside of Processor?

Hi Yi, Your observation about accessing the state stores that are already there vs. keeping state outside of Kafka Streams is a good one. We are currently working on having the state stores accessible like you mention and should be able to share some design docs shortly. Thanks Eno > On 19

Can I access Kafka Streams Key/Value store outside of Processor?

Hello, I am thinking of using the Kafka Steams feature to "unify" our real-time and scheduled workflow. An example is that in our workflow with stages A--> B --> C, the A --> B segment can be achieved in real-time, but B-->C segment is usually a done with a scheduled job, running maybe once per

kafka + logstash

Hello all I use kafka input and kafka output plugin in logstash. I have high cpu usage, what can I do to get it better? Thanks a lot

kafka + logstash

Hello all I use kafka input and kafka output plugin in logstash. I have high cpu usage, what can I do to get it better? logstash version 2.3.2 logstash-input-kafka 2.0.8 logstash-output-kafka 2.0.5 Thanks a lot

Re: Error closing Socet for ...

Looks like the producers lose the connection to the brokers. Do the brokers have enough resources to handle all the producers? Does the network support that throughput? On Sun, 19 Jun 2016, 17:27 Avi Asulin, wrote: > Hi > We are using kafka 0.8.2 with scala 2.10 version

Error closing Socet for ...

Hi We are using kafka 0.8.2 with scala 2.10 version We currently have 3 brokers and we are working with ~ 170 producers We frequently get the Error ERROR Closing socket for /170.144.181.50 because of error (kafka.network.Processor) java.io.IOException: Connection reset by peer at

Re: Wordcount with reduce

Can you show the full stack trace? How do you ingest the date into the topic? I also think, you should read the topic as KStream (instead of KTable). What de-/serializer do you specify in props. (see http://docs.confluent.io/3.0.0/streams/developer-guide.html#streams-developer-guide-serdes)

Wordcount with reduce

Hi, I want to implement wordcount example with reduce function in KTable. However, I get the error: Exception in thread "StreamThread-1" org.apache.kafka.common.errors.SerializationException: Size of data received by LongDeserializer is not 8 Here is my code: KTable

Re: test of producer's delay and consumer's delay

@jun Rao about this question，can you give me some suggestion？ > 在 2016年6月18日，上午11:26，Kafka 写道： > > hello，I have done a series of tests on kafka 0.9.0，and one of the results > confused me. > > test enviroment: > kafka cluster: 3 brokers，8core cpu ／ 8g mem ／1g netcard >

Re: consumer.poll() takes approx. 30 seconds - 0.9 new consumer api

In my tests , I am using around 24 consumer groups. I never call consumer.close() or consumer.unsubscribe() until the application is shutting down. So the consumers never leave but new consumer instances do get created as the parallel requests pile up . Also, I am reusing consumer instances