(New) reviewer field in Kafka jiras

[Joel Koshy]

I had requested infra to add a reviewer field in our jiras -
https://issues.apache.org/jira/browse/INFRA-8189 Hopefully it will
make it easier to formally keep track of a review owner for each jira.

It goes without saying that it should not be interpreted as sole
reviewer - i.e., more than one person can and should review, but I
think this is slightly better than assigning a jira back to a person
to indicate a review is required.

Thanks,

-- 
Joel

[jira] [Updated] (KAFKA-687) Rebalance algorithm should consider partitions from all topics

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-687:
-

Reviewer: Jun Rao

> Rebalance algorithm should consider partitions from all topics
> --
>
> Key: KAFKA-687
> URL: https://issues.apache.org/jira/browse/KAFKA-687
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Pablo Barrera
>Assignee: Joel Koshy
> Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, 
> KAFKA-687_2014-08-19_12:07:37.patch, KAFKA-687_2014-08-20_18:09:28.patch, 
> KAFKA-687_2014-08-25_12:36:48.patch, KAFKA-687_2014-08-28_16:20:25.patch
>
>
> The current rebalance step, as stated in the original Kafka paper [1], splits 
> the partitions per topic between all the consumers. So if you have 100 topics 
> with 2 partitions each and 10 consumers only two consumers will be used. That 
> is, for each topic all partitions will be listed and shared between the 
> consumers in the consumer group in order (not randomly).
> If the consumer group is reading from several topics at the same time it 
> makes sense to split all the partitions from all topics between all the 
> consumer. Following the example, we will have 200 partitions in total, 20 per 
> consumer, using the 10 consumers.
> The load per topic could be different and the division should consider this. 
> However even a random division should be better than the current algorithm 
> while reading from several topics and should harm reading from a few topics 
> with several partitions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-1374:
--

Reviewer: Joel Koshy
Assignee: Manikumar Reddy  (was: Joel Koshy)

> LogCleaner (compaction) does not support compressed topics
> --
>
> Key: KAFKA-1374
> URL: https://issues.apache.org/jira/browse/KAFKA-1374
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, 
> KAFKA-1374_2014-08-12_22:23:06.patch
>
>
> This is a known issue, but opening a ticket to track.
> If you try to compact a topic that has compressed messages you will run into
> various exceptions - typically because during iteration we advance the
> position based on the decompressed size of the message. I have a bunch of
> stack traces, but it should be straightforward to reproduce.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

[Todd Palino (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Palino resolved KAFKA-1588.


Resolution: Won't Fix

> Offset response does not support two requests for the same topic/partition 
> combo
> 
>
> Key: KAFKA-1588
> URL: https://issues.apache.org/jira/browse/KAFKA-1588
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Todd Palino
>Assignee: Sriharsha Chintalapani
>  Labels: newbie
>
> When performing an OffsetRequest, if you request the same topic and partition 
> combination in a single request more than once (for example, if you want to 
> get both the head and tail offsets for a partition in the same request), you 
> will get a response for both, but they will be the same offset.
> We identified that the problem is that when the offset response is assembled, 
> a map is used to store the offset info before it is converted to the response 
> format and sent to the client. Therefore, the second request for a 
> topic/partition combination will overwrite the offset from the first request.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

[Todd Palino (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114814#comment-14114814
 ] 

Todd Palino commented on KAFKA-1588:


I believe that since this involves significant changes to the protocol, and we 
have a reasonable workaround, we can close it.

> Offset response does not support two requests for the same topic/partition 
> combo
> 
>
> Key: KAFKA-1588
> URL: https://issues.apache.org/jira/browse/KAFKA-1588
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Todd Palino
>Assignee: Sriharsha Chintalapani
>  Labels: newbie
>
> When performing an OffsetRequest, if you request the same topic and partition 
> combination in a single request more than once (for example, if you want to 
> get both the head and tail offsets for a partition in the same request), you 
> will get a response for both, but they will be the same offset.
> We identified that the problem is that when the offset response is assembled, 
> a map is used to store the offset info before it is converted to the response 
> format and sent to the client. Therefore, the second request for a 
> topic/partition combination will overwrite the offset from the first request.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: Kafka-trunk #255

[Apache Jenkins Server]

See 

Changes:

[joe.stein] KAFKA-1485 Upgrade to Zookeeper 3.4.6 and create shim for ZKCLI so 
system tests can run patch by Gwen Shapira reviewed by Joe Stein and Jun Rao

--
[...truncated 747 lines...]
kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testEquality PASSED

kafka.message.MessageTest > testIsHashable PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.MessageCompressionTest > testComplexCompressDecompress PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testOffsetAssignment PASSED

kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo PASSED

kafka.producer.ProducerTest > testSendToNewTopic PASSED

kafka.producer.ProducerTest > testSendWithDeadBroker PASSED

kafka.producer.ProducerTest > testAsyncSendCanCorrectlyFailWithTimeout PASSED

kafka.producer.ProducerTest > testSendNullMessage PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.SyncProducerTest > testReachableServer PASSED

kafka.producer.SyncProducerTest > testEmptyProduceRequest PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLarge PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLargeWithAckZero PASSED

kafka.producer.SyncProducerTest > testProduceCorrectlyReceivesResponse PASSED

kafka.producer.SyncProducerTest > testProducerCanTimeout PASSED

kafka.producer.SyncProducerTest > testProduceRequestWithNoResponse PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.TopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > tooBigRequestIsRejected PASSED

kafka.network.SocketServerTest > testNullResponse PASSED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.javaapi.messag

Re: Review Request 24676: Fix KAFKA-1583

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24676/#review51855
---



core/src/main/scala/kafka/cluster/Partition.scala


Not sure we need what's in the bracket.



core/src/main/scala/kafka/server/KafkaApis.scala


a offset => an offset
request is used twice.



core/src/main/scala/kafka/server/ReplicaManager.scala


No need to explain sth that's no longer there.



core/src/main/scala/kafka/server/ReplicaManager.scala


Do we need to add the new parameter? Does it hurt to write the checkpoint 
file in unit tests?



core/src/main/scala/kafka/server/RequestPurgatory.scala


Could we make this method to return void?



core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala


What it be better to return the correct offset and just return empty 
MessageSet? The equality test can be on the offset.



core/src/main/scala/kafka/server/KafkaApis.scala


Yes, I think this should be debug level logging.



core/src/main/scala/kafka/server/RequestPurgatory.scala


need => needs



core/src/main/scala/kafka/server/RequestPurgatory.scala


a operation => an operation

It would also be good to add that "complete()" will be called exactly once.


- Jun Rao


On Aug. 27, 2014, 5 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24676/
> ---
> 
> (Updated Aug. 27, 2014, 5 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1583
> https://issues.apache.org/jira/browse/KAFKA-1583
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporated Jun's comments.
> 
> 1. I left some cases in Log since they are return values for some of their 
> APIs.
> 2. I kept the fetch info in the delayed fetch metadata since it needs to be 
> used for re-reading the log.
> 3. I kept the name of "callbackOnComplete" by following the principle that 
> only the caller knows what the callback is used for, and hence they can name 
> the callback as reponseCallback (from KafkaApi) and putCacheCallback (from 
> OffsetManager), all the callee will take the callback as "callbackOnComplete".
> 
> Unit test passed, with some other notes:
> 
> 1. Found and fix a bug in the current delayed fetch satisifaction logic: 
> previously when we calculate the bytes, we do not take in the fetchMaxBytes 
> into consideration as an upper limit for a single partition's log, but simply 
> get the diff between the current HW/LEO and the fetch offset.
> 2. Fount and fix a bug in the unit tests: we used to create replica manager 
> on the fly but did not shut it down upon completing the test, which will leak 
> the background thread (i.e. reaper thread of purgatory).
> 3. Changed the RequestPurgatory API a bit with Jun's comments. Now it has two 
> implemented functions: forceComplete() and isCompleted(), and two functions 
> that need to be instantiated in the subclasses: tryComplete() and complete(). 
> Please let me know if people have more comments on the current API.
> 4. Cleaned the SimpleFetch test, previously this test is too complicate but 
> it actually just test a simple logic of the replica manager.
> 
> One concern I have now is about the online creation of a new callback 
> function (i.e. the "def" inside the handling functions and offset manager's 
> storeOffset function, when I am running the unit test with the patch it seems 
> causing a higher CPU consumption than trunk). And could some one take a 
> another pair of eyes in running the unit tests and check the CPU performance?
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/api/FetchRequest.scala 
> 51cdccf7f90eb530cc62b094ed822b8469d50b12 
>   core/src/main/scala/kafka/api/FetchResponse.scala 
> af9308737bf7832eca018c2b3ede703f7d1209f1 
>   core/src/main/scala/kafka/api/OffsetCommitRequest.scala 
> 861a6cf11dc6b6431fcbbe9de00c74a122f204bd 
>   core/src/main/scala/kafka/api/ProducerRequest.scala 
> b2366e7eedcac17f657271d5293ff0bef6f3cbe6 
>   core/src/main/scala/kafka/api/ProducerResponse.scala 
> a286272c834b6f40164999ff8b7f8998875f2cfe 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> ff106b47e6ee194cea1cf589474fef975b9dd7e2 
>   core/src/main/scala/kafka/common/ErrorMapping.scala 
> 3fae7910e4ce17bc8325887a046f383e0c151d4

[jira] [Updated] (KAFKA-1485) Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1485:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 committed to trunk

> Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run
> --
>
> Key: KAFKA-1485
> URL: https://issues.apache.org/jira/browse/KAFKA-1485
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1.1
>Reporter: Machiel Groeneveld
>Assignee: Gwen Shapira
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1485.2.patch, KAFKA-1485.3.patch, 
> KAFKA-1485.4.patch, KAFKA-1485.patch
>
>
> I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has 
> been out for 2.5 years and seems to be ready for adoption.
> In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 
> 0.9.2 release. I can't run both versions in my tests at the same time. 
> The only compile problem I saw was in EmbeddedZookeeper.scala 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira reassigned KAFKA-1618:
---

Assignee: Gwen Shapira

> Exception thrown when running console producer with no port number for the 
> broker
> -
>
> Key: KAFKA-1618
> URL: https://issues.apache.org/jira/browse/KAFKA-1618
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>
> When running console producer with just "localhost" as the broker list, I get 
> ArrayIndexOutOfBounds exception.
> I expect either a clearer error about arguments or for the producer to 
> "guess" a default port.
> [root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 
> --broker-list localhost
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
>   at 
> kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
>   at 
> kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
>   at kafka.producer.Producer.(Producer.scala:59)
>   at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
>   at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114689#comment-14114689
 ] 

Joe Stein commented on KAFKA-1618:
--

+1 to guess a default port, would be great to make that consistent across all 
the tooled scripts, zk too

> Exception thrown when running console producer with no port number for the 
> broker
> -
>
> Key: KAFKA-1618
> URL: https://issues.apache.org/jira/browse/KAFKA-1618
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>
> When running console producer with just "localhost" as the broker list, I get 
> ArrayIndexOutOfBounds exception.
> I expect either a clearer error about arguments or for the producer to 
> "guess" a default port.
> [root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 
> --broker-list localhost
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
>   at 
> kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
>   at 
> kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
>   at kafka.producer.Producer.(Producer.scala:59)
>   at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
>   at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-1618:
---

 Summary: Exception thrown when running console producer with no 
port number for the broker
 Key: KAFKA-1618
 URL: https://issues.apache.org/jira/browse/KAFKA-1618
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira


When running console producer with just "localhost" as the broker list, I get 
ArrayIndexOutOfBounds exception.

I expect either a clearer error about arguments or for the producer to "guess" 
a default port.

[root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 --broker-list 
localhost
java.lang.ArrayIndexOutOfBoundsException: 1
at 
kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
at 
kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
at 
kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
at 
kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
at kafka.producer.Producer.(Producer.scala:59)
at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 25155: Fix KAFKA-1616

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/#review51850
---



core/src/main/scala/kafka/server/RequestPurgatory.scala


which is size => which is the size



core/src/main/scala/kafka/server/RequestPurgatory.scala


I am wonder if we should do two separate tests: (1) if enqueued() >= 
purgeInterval, we purge the dealyed queue, (2) if size >= purgeInterval, we 
purge the watchers.


- Jun Rao


On Aug. 28, 2014, 5:12 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25155/
> ---
> 
> (Updated Aug. 28, 2014, 5:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1616
> https://issues.apache.org/jira/browse/KAFKA-1616
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Purgatory size to be the sum of watched list sizes; delayed request to be the 
> expiry queue length; remove atomic integers for metrics
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala 
> ce06d2c381348deef8559374869fcaed923da1d1 
> 
> Diff: https://reviews.apache.org/r/25155/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Updated] (KAFKA-687) Rebalance algorithm should consider partitions from all topics

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-687:
-

Attachment: KAFKA-687_2014-08-28_16:20:25.patch

> Rebalance algorithm should consider partitions from all topics
> --
>
> Key: KAFKA-687
> URL: https://issues.apache.org/jira/browse/KAFKA-687
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Pablo Barrera
>Assignee: Joel Koshy
> Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, 
> KAFKA-687_2014-08-19_12:07:37.patch, KAFKA-687_2014-08-20_18:09:28.patch, 
> KAFKA-687_2014-08-25_12:36:48.patch, KAFKA-687_2014-08-28_16:20:25.patch
>
>
> The current rebalance step, as stated in the original Kafka paper [1], splits 
> the partitions per topic between all the consumers. So if you have 100 topics 
> with 2 partitions each and 10 consumers only two consumers will be used. That 
> is, for each topic all partitions will be listed and shared between the 
> consumers in the consumer group in order (not randomly).
> If the consumer group is reading from several topics at the same time it 
> makes sense to split all the partitions from all topics between all the 
> consumer. Following the example, we will have 200 partitions in total, 20 per 
> consumer, using the 10 consumers.
> The load per topic could be different and the division should consider this. 
> However even a random division should be better than the current algorithm 
> while reading from several topics and should harm reading from a few topics 
> with several partitions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114578#comment-14114578
 ] 

Joel Koshy commented on KAFKA-687:
--

Updated reviewboard https://reviews.apache.org/r/23655/
 against branch origin/trunk

> Rebalance algorithm should consider partitions from all topics
> --
>
> Key: KAFKA-687
> URL: https://issues.apache.org/jira/browse/KAFKA-687
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0
>Reporter: Pablo Barrera
>Assignee: Joel Koshy
> Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, 
> KAFKA-687_2014-08-19_12:07:37.patch, KAFKA-687_2014-08-20_18:09:28.patch, 
> KAFKA-687_2014-08-25_12:36:48.patch, KAFKA-687_2014-08-28_16:20:25.patch
>
>
> The current rebalance step, as stated in the original Kafka paper [1], splits 
> the partitions per topic between all the consumers. So if you have 100 topics 
> with 2 partitions each and 10 consumers only two consumers will be used. That 
> is, for each topic all partitions will be listed and shared between the 
> consumers in the consumer group in order (not randomly).
> If the consumer group is reading from several topics at the same time it 
> makes sense to split all the partitions from all topics between all the 
> consumer. Following the example, we will have 200 partitions in total, 20 per 
> consumer, using the 10 consumers.
> The load per topic could be different and the division should consider this. 
> However even a random division should be better than the current algorithm 
> while reading from several topics and should harm reading from a few topics 
> with several partitions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23655: Patch for KAFKA-687

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23655/
---

(Updated Aug. 28, 2014, 11:20 p.m.)


Review request for kafka.


Bugs: KAFKA-687
https://issues.apache.org/jira/browse/KAFKA-687


Repository: kafka


Description (updated)
---

v6


Diffs (updated)
-

  core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
1cf2f62ba02e4aa66bfa7575865e5d57baf82212 
  core/src/main/scala/kafka/consumer/PartitionAssignor.scala PRE-CREATION 
  core/src/main/scala/kafka/consumer/TopicCount.scala 
8b0ae5785e08272d0ea12483beae597f2eac4343 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
acfd064bdba2b031f8869011da79649efd80946f 
  core/src/main/scala/kafka/metrics/KafkaMetricsGroup.scala 
00df4621fd724826a1e79d849c762ac1c4f49868 
  core/src/main/scala/kafka/utils/ZkUtils.scala 
dcdc1ce2b02c996294e19cf480736106aaf29511 
  core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/23655/diff/


Testing
---

* I did the unit tests (including the new one) as well as mirror maker system 
test suite with roundrobin. While this is being reviewed I will run the system 
tests with symmetric


Thanks,

Joel Koshy

Re: Jira tickets for starter bugs

[Bill Bejeck]

Great

Thanks!


On Thu, Aug 28, 2014 at 7:04 PM, Joe Stein  wrote:

> It would be great if you could assign it to yourself or ask someone to
> assign it to you (comment in ticket or ask on list).  If you don't have
> access to assign to yourself we can add you as a contributor, not a problem
> just need your username for JIRA.  Having tickets assigned to you gives you
> credit for it and lets other folks know to not work on it since it is being
> worked on already.
>
> Please see
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+patch+review+tool
> in regards to how to submit patches we don't except pull requests.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
>
> On Thu, Aug 28, 2014 at 7:00 PM, Bill Bejeck  wrote:
>
> > Hi,
> >
> > For the starter bugs, do I need to have the jira ticket assigned to me,
> or
> > do I just do the work and submit a pull request?
> >
> > Thanks,
> > Bill
> >
>

Re: Jira tickets for starter bugs

[Joe Stein]

It would be great if you could assign it to yourself or ask someone to
assign it to you (comment in ticket or ask on list).  If you don't have
access to assign to yourself we can add you as a contributor, not a problem
just need your username for JIRA.  Having tickets assigned to you gives you
credit for it and lets other folks know to not work on it since it is being
worked on already.

Please see
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+patch+review+tool
in regards to how to submit patches we don't except pull requests.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Thu, Aug 28, 2014 at 7:00 PM, Bill Bejeck  wrote:

> Hi,
>
> For the starter bugs, do I need to have the jira ticket assigned to me, or
> do I just do the work and submit a pull request?
>
> Thanks,
> Bill
>

Jira tickets for starter bugs

[Bill Bejeck]

Hi,

For the starter bugs, do I need to have the jira ticket assigned to me, or
do I just do the work and submit a pull request?

Thanks,
Bill

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114390#comment-14114390
 ] 

Jun Rao commented on KAFKA-1282:


Thanks for the patch. The following should be * 100, right?

  private val connectionsLruTimeout: Long = connectionsMaxIdleMs * 1000

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Neha Narkhede
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 1282.patch, 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1617) Move Metadata Cache to TopicManager and handling of Offset Request to LogManager

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-1617:


 Summary: Move Metadata Cache to TopicManager and handling of 
Offset Request to LogManager
 Key: KAFKA-1617
 URL: https://issues.apache.org/jira/browse/KAFKA-1617
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
 Fix For: 0.9.0


This is a follow-up of KAFKA-1583. In order to make Kafka APIs a pure stateless 
layer that just forwards different requests to the corresponding managers, 
there are still two tasks left:

1. Move the metadata cache at KafkaApis to a separate manager, maybe called 
TopicManager, which will be responsible for a) handle topic metadata request, 
b) handle topic metadata update request by talking to replica manager if 
necessary.

2. Move the handling logic of offset request to the LogManager, which should 
contain all the information necessary to handle this request.

Finally, the KafkaApis class should be stateless, meaning no inner variables 
and no start()/shutdown() functions needed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 25136: KAFKA-1610-Review Request

[Mayuresh Gharat]

> On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala, line 20
> > 
> >
> > Is this necessary? ReassignedPartitionContext seems not used anywhere.

No. I suppose it got autoimported in intellij


> On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, lines 35-48
> > 
> >
> > Is this intentional?

Nope. I ran autoformat in intellij and my settings for that did this. Changing 
it now


> On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala, line 144
> > 
> >
> > Is this intentional?

Not all. Again autoformat in intellij


> On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
> > core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala, line 132
> > 
> >
> > For resulted maps that are used in the constructor parameters, as long 
> > as the constructor parameter will not change we can use mapValues.

Changing it now.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review51793
---


On Aug. 28, 2014, 6:12 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Aug. 28, 2014, 6:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Patch for replacing mapValues by map wherever necessary so that local 
> modifications to collections are not lost
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/log/LogManager.scala 
> 4d2924d04bc4bd62413edb0ee2d4aaf3c0052867 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests and everything passed and the build succeeeded
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

[jira] [Commented] (KAFKA-1612) Consumer offsets auto-commit before processing finishes

[Gian Merlino (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114320#comment-14114320
 ] 

Gian Merlino commented on KAFKA-1612:
-

I am currently working around this by setting auto.commit.enable = false and 
calling commitOffsets() manually from time to time, at certain points when I 
know that all consumed data has been fully processed. But I was hoping that 
auto.commit.enable could be made to offer guaranteed delivery for people that 
fully process each message before reading the next one. I think it doesn't 
currently, since the kafka-consumer-autocommit thread can commit the offset for 
a message at any time after kafkaStream.next() returns that message, even if it 
hasn't been fully processed yet.

Maybe instead it could commit offsets only for the previous message returned, 
instead of the current message. People that do batch processing of messages 
would still need to set auto.commit.enable = false and commitOffsets() manually 
to get guaranteed delivery, but I think that's okay, since they're doing 
something more complicated than the iterator is really able to help them with.

> Consumer offsets auto-commit before processing finishes
> ---
>
> Key: KAFKA-1612
> URL: https://issues.apache.org/jira/browse/KAFKA-1612
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1.1
>Reporter: Gian Merlino
>Assignee: Neha Narkhede
>
> In a loop like this,
>   for (message <- kafkaStream) {
>  process(message)
>   }
> The consumer can commit offsets for the next message while "process" is 
> running. If the program crashes during "process", the next run will pick up 
> from the *next* message. The message in flight at the time of the crash will 
> never actually finish processing. Instead, I would have expected the high 
> level consumer to deliver messages at least once.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1502) source jar is empty

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1502:
-

Fix Version/s: 0.8.2

> source jar is empty
> ---
>
> Key: KAFKA-1502
> URL: https://issues.apache.org/jira/browse/KAFKA-1502
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Joel Koshy
>  Labels: newbie
> Fix For: 0.8.2
>
>
> When doing a local publish, kafka_2.8.0-0.8.1.1-sources.jar only contains the 
> following files.
> META-INF/
> META-INF/MANIFEST.MF
> LICENSE
> NOTICE



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

[nicu marasoiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114248#comment-14114248
 ] 

nicu marasoiu commented on KAFKA-1282:
--

[~nehanarkhede] Hi, I implemented our discussion and applied Jun Rao 
suggestions, can you check and perhaps commit it if looks good? Hope for more 
tasks like this, do you have any suggestions?:)

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Neha Narkhede
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 1282.patch, 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1510) Force offset commits when migrating consumer offsets from zookeeper to kafka

[nicu marasoiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114244#comment-14114244
 ] 

nicu marasoiu commented on KAFKA-1510:
--

Hi, I will do this tomorrow, Friday, so I hope you will have a patch Friday 
morning your time.

So you say that the only condition to do unfiltered is kafka storage, 
regardless of dual commit mode or single commit mode, yes?

Thanks,
Nicu

> Force offset commits when migrating consumer offsets from zookeeper to kafka
> 
>
> Key: KAFKA-1510
> URL: https://issues.apache.org/jira/browse/KAFKA-1510
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Joel Koshy
>Assignee: Joel Koshy
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: kafka-1510.patch
>
>
> When migrating consumer offsets from ZooKeeper to kafka, we have to turn on 
> dual-commit (i.e., the consumers will commit offsets to both zookeeper and 
> kafka) in addition to setting offsets.storage to kafka. However, when we 
> commit offsets we only commit offsets if they have changed (since the last 
> commit). For low-volume topics or for topics that receive data in bursts 
> offsets may not move for a long period of time. Therefore we may want to 
> force the commit (even if offsets have not changed) when migrating (i.e., 
> when dual-commit is enabled) - we can add a minimum interval threshold (say 
> force commit after every 10 auto-commits) as well as on rebalance and 
> shutdown.
> Also, I think it is safe to switch the default for offsets.storage from 
> zookeeper to kafka and set the default to dual-commit (for people who have 
> not migrated yet). We have deployed this to the largest consumers at linkedin 
> and have not seen any issues so far (except for the migration caveat that 
> this jira will resolve).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 25136: KAFKA-1610-Review Request

[Guozhang Wang]

> On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
> > Thanks for the patch, some general comments: 
> > 
> > 1. In general we would like to avoid using ._1 and ._2 simply due to 
> > clarity of the code; instead we can use { case (key, value) => }.
> > 2. After thinking about it twice, I think even if the resulted collection 
> > is passed to some function as parameters, as long as we know that function 
> > will only read that value (for example 
> > ZkUtils.updatePartitionReassignmentData), but have no intention to modify 
> > it we can probably still use mapValues, which gives you the benefit of not 
> > creating one more Java collection in JVM. What do you think?
> > 3. For places we do need to use map instead of mapValues (for example in 
> > ReplicaManager when we created the delayed request's reponse status). Add 
> > comments explaning why we do so (for the above example since "acksPending" 
> > and "errorCode" may be modified after the collection is created).
> > 
> > Some detailed comments below.
> 
> Mayuresh Gharat wrote:
> I agree with point 1.)
> Regarding point 2.) I think that even if the function is only reading 
> what happens when the collection gets changed and that function reads a 
> different value. Of course if the collection is created locally and passed to 
> a function then its better to use mapValues.
> Regarding point 3.) I will add those comments

2) Yeah I agree. If the collection is used outside the current block then we 
should use map.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review51793
---


On Aug. 28, 2014, 6:12 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Aug. 28, 2014, 6:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Patch for replacing mapValues by map wherever necessary so that local 
> modifications to collections are not lost
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/log/LogManager.scala 
> 4d2924d04bc4bd62413edb0ee2d4aaf3c0052867 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests and everything passed and the build succeeeded
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

[jira] [Issue Comment Deleted] (KAFKA-1362) Publish sources and javadoc jars

[Daniel Wegener (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Wegener updated KAFKA-1362:
--

Comment: was deleted

(was: Hallo,

ich bin bis zum 18.08.2014 im Urlaub und lese meine E-Mails nur sporadisch.

Beste Grüße

Daniel Wegener
Consultant
Holisticon AG

Griegstraße 75, Haus 25
22763 Hamburg
Mobile: +49 176 1616 9025
Office: +49 40 6094 430-0
Fax:  +49 40 6094 430-30
E-Mail: daniel.wege...@holisticon.de
www.holisticon.de

Amtsgericht Hamburg, HRB 107396
Vorstand: Oliver Ihns, Dierk Harbeck
Aufsichtsrat: Sune Nilsson (Vorsitz)


)

> Publish sources and javadoc jars
> 
>
> Key: KAFKA-1362
> URL: https://issues.apache.org/jira/browse/KAFKA-1362
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Affects Versions: 0.8.1
>Reporter: Stevo Slavic
>Assignee: Joel Koshy
>  Labels: build
> Fix For: 0.8.1.1
>
> Attachments: KAFKA-1362.patch
>
>
> Currently just binaries jars get published on Maven Central (see 
> http://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.1/ ). Please 
> also publish sources and javadoc jars.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (128 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1616  Purgatory Size and Num.Delayed.Request metrics are incorrect
https://issues.apache.org/jira/browse/KAFKA-1616
KAFKA-1614  Partition log directory name and segments information exposed via 
JMX
https://issues.apache.org/jira/browse/KAFKA-1614
KAFKA-1611  Improve system test configuration
https://issues.apache.org/jira/browse/KAFKA-1611
KAFKA-1610  Local modifications to collections generated from mapValues will be 
lost
https://issues.apache.org/jira/browse/KAFKA-1610
KAFKA-1604  System Test for Transaction Management
https://issues.apache.org/jira/browse/KAFKA-1604
KAFKA-1601  ConsoleConsumer/SimpleConsumerPerformance should be 
transaction-aware
https://issues.apache.org/jira/browse/KAFKA-1601
KAFKA-1600  Controller failover not working correctly.
https://issues.apache.org/jira/browse/KAFKA-1600
KAFKA-1597  New metrics: ResponseQueueSize and BeingSentResponses
https://issues.apache.org/jira/browse/KAFKA-1597
KAFKA-1586  support sticky partitioning in the new producer
https://issues.apache.org/jira/browse/KAFKA-1586
KAFKA-1585  Client: Infinite "conflict in /consumers/"
https://issues.apache.org/jira/browse/KAFKA-1585
KAFKA-1583  Kafka API Refactoring
https://issues.apache.org/jira/browse/KAFKA-1583
KAFKA-1577  Exception in ConnectionQuotas while shutting down
https://issues.apache.org/jira/browse/KAFKA-1577
KAFKA-1569  Tool for performance and correctness of transactions end-to-end
https://issues.apache.org/jira/browse/KAFKA-1569
KAFKA-1561  Data Loss for Incremented Replica Factor and Leader Election
https://issues.apache.org/jira/browse/KAFKA-1561
KAFKA-1543  Changing replication factor
https://issues.apache.org/jira/browse/KAFKA-1543
KAFKA-1541   Add transactional request definitions to clients package
https://issues.apache.org/jira/browse/KAFKA-1541
KAFKA-1528  Normalize all the line endings
https://issues.apache.org/jira/browse/KAFKA-1528
KAFKA-1527  SimpleConsumer should be transaction-aware
https://issues.apache.org/jira/browse/KAFKA-1527
KAFKA-1526  Producer performance tool should have an option to enable 
transactions
https://issues.apache.org/jira/browse/KAFKA-1526
KAFKA-1525  DumpLogSegments should print transaction IDs
https://issues.apache.org/jira/browse/KAFKA-1525
KAFKA-1524  Implement transactional producer
https://issues.apache.org/jira/browse/KAFKA-1524
KAFKA-1523  Implement transaction manager module
https://issues.apache.org/jira/browse/KAFKA-1523
KAFKA-1517  Messages is a required argument to Producer Performance Test
https://issues.apache.org/jira/browse/KAFKA-1517
KAFKA-1510  Force offset commits when migrating consumer offsets from zookeeper 
to kafka
https://issues.apache.org/jira/browse/KAFKA-1510
KAFKA-1509  Restart of destination broker after unreplicated partition move 
leaves partitions without leader
https://issues.apache.org/jira/browse/KAFKA-1509
KAFKA-1507  Using GetOffsetShell against non-existent topic creates the topic 
unintentionally
https://issues.apache.org/jira/browse/KAFKA-1507
KAFKA-1500  adding new consumer requests using the new protocol
https://issues.apache.org/jira/browse/KAFKA-1500
KAFKA-1499  Broker-side compression configuration
https://issues.apache.org/jira/browse/KAFKA-1499
KAFKA-1498  new producer performance and bug improvements
https://issues.apache.org/jira/browse/KAFKA-1498
KAFKA-1496  Using batch message in sync producer only sends the first message 
if we use a Scala Stream as the argument 
https://issues.apache.org/jira/browse/KAFKA-1496
KAFKA-1485  Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system 
tests can run
https://issues.apache.org/jira/browse/KAFKA-1485
KAFKA-1481  Stop using dashes AND underscores as separators in MBean names
https://issues.apache.org/jira/browse/KAFKA-1481
KAFKA-1477  add authentication layer and initial JKS x509 implementation for 
brokers, producers and consumer for network communication
https://issues.apache.org/jira/browse/KAFKA-1477
KAFKA-1476  Get a list of consumer groups
https://issues.apache.org/jira/browse/KAFKA-1476
KAFKA-1475  Kafka consumer stops LeaderFinder/FetcherThreads, but application 
does not know
https://issues.apache.org/jira/browse/KAFKA-1475
KAFKA-1471  Add Producer Unit Tests for LZ4 and LZ4HC compression
https://issues.apache.org/jira/browse/KAFKA-1471
KAFKA-1468  Improve perf tests
https://issues.apache.org/jira/browse/KAFKA-1468
KAFKA-1460  NoReplicaOnlineException: No rep

Re: Review Request 25136: KAFKA-1610-Review Request

[Mayuresh Gharat]

> On Aug. 28, 2014, 5:36 p.m., Guozhang Wang wrote:
> > Thanks for the patch, some general comments: 
> > 
> > 1. In general we would like to avoid using ._1 and ._2 simply due to 
> > clarity of the code; instead we can use { case (key, value) => }.
> > 2. After thinking about it twice, I think even if the resulted collection 
> > is passed to some function as parameters, as long as we know that function 
> > will only read that value (for example 
> > ZkUtils.updatePartitionReassignmentData), but have no intention to modify 
> > it we can probably still use mapValues, which gives you the benefit of not 
> > creating one more Java collection in JVM. What do you think?
> > 3. For places we do need to use map instead of mapValues (for example in 
> > ReplicaManager when we created the delayed request's reponse status). Add 
> > comments explaning why we do so (for the above example since "acksPending" 
> > and "errorCode" may be modified after the collection is created).
> > 
> > Some detailed comments below.

I agree with point 1.)
Regarding point 2.) I think that even if the function is only reading what 
happens when the collection gets changed and that function reads a different 
value. Of course if the collection is created locally and passed to a function 
then its better to use mapValues.
Regarding point 3.) I will add those comments


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review51793
---


On Aug. 28, 2014, 6:12 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Aug. 28, 2014, 6:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Patch for replacing mapValues by map wherever necessary so that local 
> modifications to collections are not lost
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/log/LogManager.scala 
> 4d2924d04bc4bd62413edb0ee2d4aaf3c0052867 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests and everything passed and the build succeeeded
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

Re: Review Request 23655: Patch for KAFKA-687

[Joel Koshy]

> On Aug. 28, 2014, 1:32 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala, line 
> > 721
> > 
> >
> > Could we use case to avoid _._1 ?
> > 
> > Also, could you explain why we want to call .view on a map?

I probably won't put a comment in the code since it is standard scala. But the 
view helps delay the materialization of the collection until actually 
needed/forced either explicitly or some forcing operation. I did some toy tests 
with the above collection to verify.


- Joel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23655/#review51694
---


On Aug. 25, 2014, 7:36 p.m., Joel Koshy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23655/
> ---
> 
> (Updated Aug. 25, 2014, 7:36 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-687
> https://issues.apache.org/jira/browse/KAFKA-687
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> v5
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 
> d97962d3840179b1abf01459522c58e59102ac8d 
>   core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
> 1cf2f62ba02e4aa66bfa7575865e5d57baf82212 
>   core/src/main/scala/kafka/consumer/PartitionAssignor.scala PRE-CREATION 
>   core/src/main/scala/kafka/consumer/TopicCount.scala 
> 8b0ae5785e08272d0ea12483beae597f2eac4343 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> acfd064bdba2b031f8869011da79649efd80946f 
>   core/src/main/scala/kafka/metrics/KafkaMetricsGroup.scala 
> 00df4621fd724826a1e79d849c762ac1c4f49868 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> b8698ee1469c8fbc92ccc176d916eb3e28b87867 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
>   core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/23655/diff/
> 
> 
> Testing
> ---
> 
> * I did the unit tests (including the new one) as well as mirror maker system 
> test suite with roundrobin. While this is being reviewed I will run the 
> system tests with symmetric
> 
> 
> Thanks,
> 
> Joel Koshy
> 
>

Re: Review Request 23655: Patch for KAFKA-687

[Joel Koshy]

> On Aug. 28, 2014, 1:32 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/consumer/ConsumerConfig.scala, line 52
> > 
> >
> > This probably should be named DefaultPartitionAssignmentStrategy.

Ok - it seems a number of defaults are improperly named.


> On Aug. 28, 2014, 1:32 a.m., Jun Rao wrote:
> > core/src/main/scala/kafka/consumer/PartitionAssignor.scala, line 47
> > 
> >
> > partitionAssginment is a bit confusing now that we have the partition 
> > assignor. Perhaps both the val and the method name in ZkUtils should be 
> > called replicaAssignmentPerTopic.

Agreed - there is already another replicaAssignmentPerTopic utility in ZkUtils, 
but it turns out I can actually use ZkUtils.getPartitionsForTopic.


> On Aug. 28, 2014, 1:32 a.m., Jun Rao wrote:
> > core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala, lines 
> > 214-222
> > 
> >
> > I am not sure if this is necessary. The input is a map with 
> > topicAndPartition as the key. This implies that each partition has only 1 
> > owner.

That's very true - dumb mistake. I actually do want to check for conflicts so I 
refactored the test a bit and managed to get rid of the "forConsumer" API 
version of the roundrobin assignor.


- Joel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23655/#review51694
---


On Aug. 25, 2014, 7:36 p.m., Joel Koshy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23655/
> ---
> 
> (Updated Aug. 25, 2014, 7:36 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-687
> https://issues.apache.org/jira/browse/KAFKA-687
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> v5
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 
> d97962d3840179b1abf01459522c58e59102ac8d 
>   core/src/main/scala/kafka/consumer/ConsumerConfig.scala 
> 1cf2f62ba02e4aa66bfa7575865e5d57baf82212 
>   core/src/main/scala/kafka/consumer/PartitionAssignor.scala PRE-CREATION 
>   core/src/main/scala/kafka/consumer/TopicCount.scala 
> 8b0ae5785e08272d0ea12483beae597f2eac4343 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> acfd064bdba2b031f8869011da79649efd80946f 
>   core/src/main/scala/kafka/metrics/KafkaMetricsGroup.scala 
> 00df4621fd724826a1e79d849c762ac1c4f49868 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 
> b8698ee1469c8fbc92ccc176d916eb3e28b87867 
>   core/src/main/scala/kafka/utils/ZkUtils.scala 
> dcdc1ce2b02c996294e19cf480736106aaf29511 
>   core/src/test/scala/unit/kafka/consumer/PartitionAssignorTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/23655/diff/
> 
> 
> Testing
> ---
> 
> * I did the unit tests (including the new one) as well as mirror maker system 
> test suite with roundrobin. While this is being reviewed I will run the 
> system tests with symmetric
> 
> 
> Thanks,
> 
> Joel Koshy
> 
>

Re: Review Request 25044: KAFKA-1611 - Improve system test configuration

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25044/#review51803
---

Ship it!


Thanks for the patch. LGTM.

- Guozhang Wang


On Aug. 26, 2014, 12:04 a.m., Gwen Shapira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25044/
> ---
> 
> (Updated Aug. 26, 2014, 12:04 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> make the config a bit more "out of the box" for the common case of local 
> cluster. This includes:
> 1. Fix cluster_config.json of migration testsuite, it has hardcoded path that 
> prevents it from working out of the box at all.
> 2. Use JAVA_HOME environment variable if "default" is specified and if 
> JAVA_HOME is defined. The current guessing method is a bit broken and using 
> JAVA_HOME will allow devs to configure their default java dir without editing 
> multiple cluster_config.json files in multiple places.
> 
> 
> Diffs
> -
> 
>   system_test/migration_tool_testsuite/cluster_config.json 8353e56 
>   system_test/utils/system_test_utils.py 50340f0 
> 
> Diff: https://reviews.apache.org/r/25044/diff/
> 
> 
> Testing
> ---
> 
> Running system tests bunch of times :)
> 
> 
> Thanks,
> 
> Gwen Shapira
> 
>

Re: Review Request 25136: KAFKA-1610-Review Request

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/
---

(Updated Aug. 28, 2014, 6:12 p.m.)


Review request for kafka.


Bugs: KAFKA-1610
https://issues.apache.org/jira/browse/KAFKA-1610


Repository: kafka


Description
---

Patch for replacing mapValues by map wherever necessary so that local 
modifications to collections are not lost


Diffs
-

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
691d69a49a240f38883d2025afaec26fd61281b5 
  core/src/main/scala/kafka/controller/KafkaController.scala 
8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
  core/src/main/scala/kafka/log/LogManager.scala 
4d2924d04bc4bd62413edb0ee2d4aaf3c0052867 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
e0f14e25af03e6d4344386dcabc1457ee784d345 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
9481508fc2d6140b36829840c337e557f3d090da 
  core/src/main/scala/kafka/server/KafkaApis.scala 
c584b559416b3ee4bcbec5966be4891e0a03eefb 
  core/src/main/scala/kafka/server/KafkaServer.scala 
28711182aaa70eaa623de858bc063cb2613b2a4d 
  core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
af4783646803e58714770c21f8c3352370f26854 
  core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 

Diff: https://reviews.apache.org/r/25136/diff/


Testing (updated)
---

Ran the unit tests and everything passed and the build succeeeded


Thanks,

Mayuresh Gharat

[jira] [Commented] (KAFKA-1420) Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with TestUtils.createTopic in unit tests

[Jonathan Natkins (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114079#comment-14114079
 ] 

Jonathan Natkins commented on KAFKA-1420:
-

Wanted to bump this ticket so the patch doesn't get irreparably stale.

> Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK with 
> TestUtils.createTopic in unit tests
> --
>
> Key: KAFKA-1420
> URL: https://issues.apache.org/jira/browse/KAFKA-1420
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Jun Rao
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1420.patch, KAFKA-1420_2014-07-30_11:18:26.patch, 
> KAFKA-1420_2014-07-30_11:24:55.patch, KAFKA-1420_2014-08-02_11:04:15.patch, 
> KAFKA-1420_2014-08-10_14:12:05.patch, KAFKA-1420_2014-08-10_23:03:46.patch
>
>
> This is a follow-up JIRA from KAFKA-1389.
> There are a bunch of places in the unit tests where we misuse 
> AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK to create topics, 
> where TestUtils.createTopic needs to be used instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 25136: KAFKA-1610-Review Request

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review51793
---


Thanks for the patch, some general comments: 

1. In general we would like to avoid using ._1 and ._2 simply due to clarity of 
the code; instead we can use { case (key, value) => }.
2. After thinking about it twice, I think even if the resulted collection is 
passed to some function as parameters, as long as we know that function will 
only read that value (for example ZkUtils.updatePartitionReassignmentData), but 
have no intention to modify it we can probably still use mapValues, which gives 
you the benefit of not creating one more Java collection in JVM. What do you 
think?
3. For places we do need to use map instead of mapValues (for example in 
ReplicaManager when we created the delayed request's reponse status). Add 
comments explaning why we do so (for the above example since "acksPending" and 
"errorCode" may be modified after the collection is created).

Some detailed comments below.


core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala


Is this necessary? ReassignedPartitionContext seems not used anywhere.



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Is this intentional?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Is this intentional?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


Could we use a new line for the nested map?



core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala


For resulted maps that are used in the constructor parameters, as long as 
the constructor parameter will not change we can use mapValues.


- Guozhang Wang


On Aug. 28, 2014, 2:28 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Aug. 28, 2014, 2:28 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Patch for replacing mapValues by map wherever necessary so that local 
> modifications to collections are not lost
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/log/LogManager.scala 
> 4d2924d04bc4bd62413edb0ee2d4aaf3c0052867 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

[jira] [Updated] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1616:
-

Attachment: KAFKA-1616_2014-08-28_10:12:17.patch

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch, KAFKA-1616_2014-08-28_10:12:17.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113984#comment-14113984
 ] 

Guozhang Wang commented on KAFKA-1616:
--

Updated reviewboard https://reviews.apache.org/r/25155/
 against branch origin/trunk

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch, KAFKA-1616_2014-08-28_10:12:17.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1616:
-

Status: Patch Available  (was: Open)

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1616:
-

Attachment: KAFKA-1616.patch

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113981#comment-14113981
 ] 

Guozhang Wang commented on KAFKA-1616:
--

Created reviewboard https://reviews.apache.org/r/25155/
 against branch origin/trunk

> Purgatory Size and Num.Delayed.Request metrics are incorrect
> 
>
> Key: KAFKA-1616
> URL: https://issues.apache.org/jira/browse/KAFKA-1616
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
> Attachments: KAFKA-1616.patch
>
>
> The request purgatory used two atomic integers "watched" and "unsatisfied" to 
> record the purgatory size ( = watched + unsatisfied) and number of delayed 
> requests ( = unsatisfied). But due to some race conditions these two atomic 
> integers are not updated correctly, result in incorrect metrics.
> Proposed solution: to have a cleaner semantics, we can define the "purgatory 
> size" to be just the number of elements in the watched lists, and the "number 
> of delayed requests" to be just the length of the expiry queue. And instead 
> of using two atomic integeres we just compute the size of the lists / queue 
> on the fly each time the metrics are pulled. This may use some more CPU 
> cycles for these two metrics but should be minor, and the correctness is 
> guaranteed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 25155: Fix KAFKA-1616

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/
---

(Updated Aug. 28, 2014, 5:12 p.m.)


Review request for kafka.


Bugs: KAFKA-1616
https://issues.apache.org/jira/browse/KAFKA-1616


Repository: kafka


Description
---

Purgatory size to be the sum of watched list sizes; delayed request to be the 
expiry queue length; remove atomic integers for metrics


Diffs (updated)
-

  core/src/main/scala/kafka/server/RequestPurgatory.scala 
ce06d2c381348deef8559374869fcaed923da1d1 

Diff: https://reviews.apache.org/r/25155/diff/


Testing
---


Thanks,

Guozhang Wang

Review Request 25155: Fix KAFKA-1616

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25155/
---

Review request for kafka.


Bugs: KAFKA-1616
https://issues.apache.org/jira/browse/KAFKA-1616


Repository: kafka


Description
---

Purgatory size to be the sum of watched list sizes; delayed request to be the 
expiry queue length; remove atomic integers for metrics


Diffs
-

  core/src/main/scala/kafka/server/RequestPurgatory.scala 
ce06d2c381348deef8559374869fcaed923da1d1 

Diff: https://reviews.apache.org/r/25155/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Created] (KAFKA-1616) Purgatory Size and Num.Delayed.Request metrics are incorrect

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-1616:


 Summary: Purgatory Size and Num.Delayed.Request metrics are 
incorrect
 Key: KAFKA-1616
 URL: https://issues.apache.org/jira/browse/KAFKA-1616
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.9.0


The request purgatory used two atomic integers "watched" and "unsatisfied" to 
record the purgatory size ( = watched + unsatisfied) and number of delayed 
requests ( = unsatisfied). But due to some race conditions these two atomic 
integers are not updated correctly, result in incorrect metrics.

Proposed solution: to have a cleaner semantics, we can define the "purgatory 
size" to be just the number of elements in the watched lists, and the "number 
of delayed requests" to be just the length of the expiry queue. And instead of 
using two atomic integeres we just compute the size of the lists / queue on the 
fly each time the metrics are pulled. This may use some more CPU cycles for 
these two metrics but should be minor, and the correctness is guaranteed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113641#comment-14113641
 ] 

Otis Gospodnetic commented on KAFKA-1481:
-

bq. if you know the clientId, you can parse the rest of the string. 

I think this makes parsers unnecessarily difficult to write and probably harder 
to maintain.  Parsers have to remember things this way.

bq. If this patch is going to cause existing users that are not having a 
problem issues 

I think it won't cause them issues.  I *think* they would just need to change 
which delimiter character they use to break MBean names into parts.

bq. If it is causing existing users issues by not patching it

I think it is causing issues to anyone who has any of the multiple characters 
Kafka uses as delimiters in their hostnames or topic names.  So I think these 
people have issues when they first try parsing stuff from JMX, and then they 
have to go modify their hostnames, topic names and such.


> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>  Labels: patch
> Fix For: 0.8.2
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)