date:20140115

[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

2014-01-15 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872210#comment-13872210
 ] 

Jun Rao commented on KAFKA-1196:


Yes, I wasn't thinking of fixing the issue since I can't think of an easy way. 
I was just thinking of giving the client the right error message so that the 
client is not confused  btw this case and an actual log corruption.

 java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
 ---

 Key: KAFKA-1196
 URL: https://issues.apache.org/jira/browse/KAFKA-1196
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.0
 Environment: running java 1.7, linux and kafka compiled against scala 
 2.9.2
Reporter: Gerrit Jansen van Vuuren
Assignee: Neha Narkhede
Priority: Blocker
  Labels: newbie
 Fix For: 0.9.0


 I have 6 topics each with 8 partitions spread over 4 kafka servers.
 the servers are 24 core 72 gig ram.
 While consuming from the topics I get an IlegalArgumentException and all 
 consumption stops, the error keeps on throwing.
 I've tracked it down to FectchResponse.scala line 33
 The error happens when the FetchResponsePartitionData object's readFrom 
 method calls:
 messageSetBuffer.limit(messageSetSize)
 I put in some debug code the the messageSetSize is 671758648, while the 
 buffer.capacity() gives 155733313, for some reason the buffer is smaller than 
 the required message size.
 I don't know the consumer code enough to debug this. It doesn't matter if 
 compression is used or not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

2014-01-15 Thread Gerrit Jansen van Vuuren (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872215#comment-13872215
]

Gerrit Jansen van Vuuren commented on KAFKA-1196:
-

:) , it doesn't seem like there is an easy fix,
I'm trying to write my own consumer that would try and work around this error,
i.e. ignore the initial message length, and still read the message sets, even
though the actual topic/partition/messageset sequences might go over the
initial 4 byte int message size. will comment on how this goes.

java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
---

Key: KAFKA-1196
URL: https://issues.apache.org/jira/browse/KAFKA-1196
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.8.0
Environment: running java 1.7, linux and kafka compiled against scala
2.9.2
Reporter: Gerrit Jansen van Vuuren
Assignee: Neha Narkhede
Priority: Blocker
Labels: newbie
Fix For: 0.9.0

I have 6 topics each with 8 partitions spread over 4 kafka servers.
the servers are 24 core 72 gig ram.
While consuming from the topics I get an IlegalArgumentException and all
consumption stops, the error keeps on throwing.
I've tracked it down to FectchResponse.scala line 33
The error happens when the FetchResponsePartitionData object's readFrom
method calls:
messageSetBuffer.limit(messageSetSize)
I put in some debug code the the messageSetSize is 671758648, while the
buffer.capacity() gives 155733313, for some reason the buffer is smaller than
the required message size.
I don't know the consumer code enough to debug this. It doesn't matter if
compression is used or not.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (KAFKA-1202) optimize ZK access in KafkaController

2014-01-15 Thread Jun Rao (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1202.


   Resolution: Fixed
Fix Version/s: 0.8.1

Thanks for the review. Committed to trunk. Also, incorporated changes in 
kafka-1020.

 optimize ZK access in KafkaController
 -

 Key: KAFKA-1202
 URL: https://issues.apache.org/jira/browse/KAFKA-1202
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
Reporter: Jun Rao
Assignee: Jun Rao
 Fix For: 0.8.1

 Attachments: KAFKA-1202.patch


 In KafkaController, we access ZK in the following places. Those accesses are 
 not necessary since we can read from the cache in the controller.
 In onBrokerFailure(deadBrokers: Seq[Int]),
 replicaStateMachine.handleStateChanges(getAllReplicasOnBroker(zkClient, 
 controllerContext.allTopics.toSeq, deadBrokers), OfflineReplica)
   }
 In onBrokerStartup(newBrokers: Seq[Int])
 replicaStateMachine.handleStateChanges(getAllReplicasOnBroker(zkClient, 
 controllerContext.allTopics.toSeq, newBrokers), OnlineReplica)
   }
 In  shutdownBroker(),
 getPartitionsAssignedToBroker(zkClient, controllerContext.allTopics.toSeq, 
 id).map {
   }



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (KAFKA-1020) Remove getAllReplicasOnBroker from KafkaController

2014-01-15 Thread Jun Rao (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1020:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Fixed as part of kafka-1202.

 Remove getAllReplicasOnBroker from KafkaController
 --

 Key: KAFKA-1020
 URL: https://issues.apache.org/jira/browse/KAFKA-1020
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.8.1

 Attachments: KAFKA-1020.v1.patch


 Today KafkaController call getAllReplicasOnBroker on broker failure and new 
 broker start up to get all the replicas that broker is holding (or suppose to 
 hold). This function actually issue a read on each topic's partition znodes. 
 With large number of topic/partitions this could seriously increase the 
 latency of handling broker failure and new broker startup.
 On the other hand, ControllerContext maintains a partitionReplicaAssignment 
 cache, which is designed to keep the most updated partition replica 
 assignment according to ZK. So instead of reading from ZK, we could just read 
 from the local cache, given that partitionReplicaAssignment is guaranteed to 
 be up-to-date.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (KAFKA-1196) java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

2014-01-15 Thread Gerrit Jansen van Vuuren (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872281#comment-13872281
 ] 

Gerrit Jansen van Vuuren commented on KAFKA-1196:
-

I've written a consumer https://github.com/gerritjvv/kafka-fast, its running on 
the same dataset implemented what I said in the previous comment, I do not see 
any errors yet.


 java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
 ---

 Key: KAFKA-1196
 URL: https://issues.apache.org/jira/browse/KAFKA-1196
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.0
 Environment: running java 1.7, linux and kafka compiled against scala 
 2.9.2
Reporter: Gerrit Jansen van Vuuren
Assignee: Neha Narkhede
Priority: Blocker
  Labels: newbie
 Fix For: 0.9.0


 I have 6 topics each with 8 partitions spread over 4 kafka servers.
 the servers are 24 core 72 gig ram.
 While consuming from the topics I get an IlegalArgumentException and all 
 consumption stops, the error keeps on throwing.
 I've tracked it down to FectchResponse.scala line 33
 The error happens when the FetchResponsePartitionData object's readFrom 
 method calls:
 messageSetBuffer.limit(messageSetSize)
 I put in some debug code the the messageSetSize is 671758648, while the 
 buffer.capacity() gives 155733313, for some reason the buffer is smaller than 
 the required message size.
 I don't know the consumer code enough to debug this. It doesn't matter if 
 compression is used or not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Kafka server start from resource negotiator(s)

2014-01-15 Thread Joe Stein

Hi, I am working at adding some code to the kafka.Kafka.scala file.

What I was thinking for the Utils.loadProps is to create some
ResourceNegotiatedConfiguration class or something to sit with it so
various frameworks can be supported.

For Mesos I am thinking of storing the properties in zookeeper. For a new
node joining = registering a default config (originally set by a tool) and
saving it unique for that broker in a znode.  This znode would be the input
to the serverConfig for the KafkaServerStartable.

I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos tool too.
 that tool will take in a server.properties file to set the original
default. Once loaded in zookeeper another command to take a property and
flag it with a function to run (like the use new broker id value) and
such.

I would rather have some implementation in Kafka.scala and have object
KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts folks
might have?

I will be creating JIRA soon for it.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/

40 matches

Mail list logo