[ https://issues.apache.org/jira/browse/KAFKA-188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484654#comment-13484654 ]
Neha Narkhede commented on KAFKA-188: ------------------------------------- Took a quick look at patch v4. Here are few review comments - 1. KafkaConfig We should probably raise an error if the same log directory name was specified more than once. 2. LogManager 2.1. I see you have a check in createLogIfNotExists to return if the log is already created. I guess this will happen if two threads execute the following at the same time and enter createLogIfNotExists one after the other. logs.get(topicAndPartition) match { case null => createLogIfNotExists(topicAndPartition) I wonder if it is useful to move the lock to getOrCreateLog instead ? Also, shouldn't we use the same lock to protect other accesses to the "logs" data structure (getLog(), allLogs() and topics()) ? 2.2. Fix typo on nextLogDir "chose the" -> "choose the" 3. ReplicaManager 3.1 Does it make sense to handle the absence of a matching Log for a topic partition correctly, instead of assuming the presence of one through the get API on an Option ? 3.2 Nit pick -> "highwater marks" -> "high watermarks" or "highwatermarks" ? :) 4. HighwatermarkCheckpoint 4.1 While you're in there, do you mind changing the following API to take in a map of TopicAndPartition->Long instead ? We've been bitten by scala bugs that don't handle equality on tuples very well. def write(highwaterMarksPerPartition: Map[(String, Int), Long]) > Support multiple data directories > --------------------------------- > > Key: KAFKA-188 > URL: https://issues.apache.org/jira/browse/KAFKA-188 > Project: Kafka > Issue Type: New Feature > Reporter: Jay Kreps > Attachments: KAFKA-188.patch, KAFKA-188-v2.patch, KAFKA-188-v3.patch, > KAFKA-188-v4.patch > > > Currently we allow only a single data directory. This means that a multi-disk > configuration needs to be a RAID array or LVM volume or something like that > to be mounted as a single directory. > For a high-throughput low-reliability configuration this would mean RAID0 > striping. Common wisdom in Hadoop land has it that a JBOD setup that just > mounts each disk as a separate directory and does application-level balancing > over these results in about 30% write-improvement. For example see this claim > here: > http://old.nabble.com/Re%3A-RAID-vs.-JBOD-p21466110.html > It is not clear to me why this would be the case--it seems the RAID > controller should be able to balance writes as well as the application so it > may depend on the details of the setup. > Nonetheless this would be really easy to implement, all you need to do is add > multiple data directories and balance partition creation over these disks. > One problem this might cause is if a particular topic is much larger than the > others it might unbalance the load across the disks. The partition->disk > assignment policy should probably attempt to evenly spread each topic to > avoid this, rather than just trying keep the number of partitions balanced > between disks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira