Re: Consumer keeps looking connection

2014-11-02 Thread Chen Wang 
When I check the server log, it also flushes with exception:

[2014-11-02 10:00:32,284] ERROR Closing socket for /10.93.80.119 because of
error (kafka.network.Processor)
java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at
sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565)
at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147)
at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:69)
at kafka.network.MultiSend.writeTo(Transmission.scala:102)
at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:124)
at kafka.network.MultiSend.writeTo(Transmission.scala:102)
at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:219)
at kafka.network.Processor.write(SocketServer.scala:405)
at kafka.network.Processor.run(SocketServer.scala:265)
at java.lang.Thread.run(Thread.java:744)

On Sat, Nov 1, 2014 at 9:46 PM, Chen Wang chen.apache.s...@gmail.com
wrote:

 Hello Folks,
 I am using Highlevel consumer, and it seems to drop connections
 intermittently:

 2014-11-01 13:34:40 SimpleConsumer [INFO] Reconnect due to socket error:
 Received -1 when reading from channel, socket has likely been closed.
 2014-11-01 13:34:40 ConsumerFetcherThread [WARN]
 [ConsumerFetcherThread-campaign_open_consumer_targeting_20141031_trgt-storm03-1414801367127-40cc618a-0-5],
 Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 20220;
 ClientId:
 campaign_open_consumer_targeting_20141031-ConsumerFetcherThread-campaign_open_consumer_targeting_20141031_trgt-storm03-1414801367127-40cc618a-0-5;
 ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo:
 [test_topic,18] - PartitionFetchInfo(1681313989,4194304),[test_topic,21]
 - PartitionFetchInfo(141266339,4194304)
 java.net.ConnectException: Connection refused
 at sun.nio.ch.Net.connect(Native Method)
 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
 at kafka.network.BlockingChannel.connect(Unknown Source)
 at kafka.consumer.SimpleConsumer.connect(Unknown Source)
 at kafka.consumer.SimpleConsumer.reconnect(Unknown Source)
 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source)
 at
 kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown
 Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown
 Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
 Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
 Source)
 at kafka.metrics.KafkaTimer.time(Unknown Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown Source)
 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown
 Source)
 at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown
 Source)
 at kafka.metrics.KafkaTimer.time(Unknown Source)
 at kafka.consumer.SimpleConsumer.fetch(Unknown Source)
 at kafka.server.AbstractFetcherThread.processFetchRequest(Unknown
 Source)
 at kafka.server.AbstractFetcherThread.doWork(Unknown Source)
 at kafka.utils.ShutdownableThread.run(Unknown Source)
 2014-11-01 13:34:40 VerifiableProperties [INFO] Verifying properties

 or sometimes:
 2014-11-01 13:34:40 SimpleConsumer [INFO] Reconnect due to socket error:
 null
 2014-11-01 13:34:40 ConsumerFetcherThread [WARN]
 [ConsumerFetcherThread-campaign_open_consumer_targeting_20141031_trgt-storm03-1414801367127-40cc618a-0-5],
 Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 20222;
 ClientId:
 campaign_open_consumer_targeting_20141031-ConsumerFetcherThread-campaign_open_consumer_targeting_20141031_trgt-storm03-1414801367127-40cc618a-0-5;
 ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo:
 [test_topic,18] - PartitionFetchInfo(1681313989,4194304)
 java.net.ConnectException: Connection refused
 at sun.nio.ch.Net.connect(Native Method)
 at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
 at kafka.network.BlockingChannel.connect(Unknown Source)
 at kafka.consumer.SimpleConsumer.connect(Unknown Source)
 at kafka.consumer.SimpleConsumer.reconnect(Unknown Source)
 at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source)
 at
 kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown
 Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown
 Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
 Source)
 at
 kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown
 Source)
 at

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-11-02 Thread Erik van Oosten 

Hi Jun,

The quotes are because of a regression in Metrics 2.2.0. IMHO Metrics 
2.2.0 should not be used because of this. Just downgrade to Metrics 
2.1.5 and you are good.


Of course, upgrading to Metrics 3 would do the trick also.

Kind regards,
Erik.


Jun Rao schreef op 17-10-14 om 20:54:

Hi, everyone,

We are fixing the mbean names in kafka-1482, by adding separate explicit
tags in the name for things like clientId and topic. Another thing that
some people have complained before is that we use quotes in the jmx name.
Should we also just get rid of the quotes as part of kafka-1482? So,
instead of
kafka.server:type=BrokerTopicMetrics,name=topic-1-BytesInPerSec
we will have
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1

Thanks,

Jun




--
Erik van Oosten
http://www.day-to-day-stuff.blogspot.com/