[
https://issues.apache.org/jira/browse/KAFKA-188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Kreps updated KAFKA-188:
----------------------------
Attachment: KAFKA-188-v5.patch
New patch, rebased and addresses Neha's comments:
1. Good thought. Added a check in LogManager to detect duplicate data
directories. This is not the only bad possibility though. It is possible to
have another kafka process that has opened using the same data directory. I am
not sure what would happen, but something bad. I added a per data-directory
file lock to check for this. This adds a new file .lock to each data directory
and uses it to do the equivalent of flock/funlock. This will lock access across
processes or within a process.
2.1. I agree this is a bit roundabout. The reason is that the logs are in a
Pool, which is really a ConcurrentHashMap. These are nice as they don't lock
the whole hash table on each lookup. So I think although it is a little more
verbose it is better how it is because in the common case (fetching a log)
there is no global lock needed. This should also make the other accesses
threadsafe.
2.2. Learned to spell Choose. :-)
3.1. I don't really understand this code that well, so I am not sure. If it is
a programming error to for there not to be a log present then I would rather
leave it (I think you would get the NoSuchElementException and it would be
clear what happened). The reason is that adding a match/case statement in the
middle of that groupby is going to make it awfully hard to understand.
3.2. Fixed, nice catch.
4.1. Done.
Also:
1. Re-arranged methods in LogManager to make a little more sense.
> Support multiple data directories
> ---------------------------------
>
> Key: KAFKA-188
> URL: https://issues.apache.org/jira/browse/KAFKA-188
> Project: Kafka
> Issue Type: New Feature
> Reporter: Jay Kreps
> Attachments: KAFKA-188.patch, KAFKA-188-v2.patch, KAFKA-188-v3.patch,
> KAFKA-188-v4.patch, KAFKA-188-v5.patch
>
>
> Currently we allow only a single data directory. This means that a multi-disk
> configuration needs to be a RAID array or LVM volume or something like that
> to be mounted as a single directory.
> For a high-throughput low-reliability configuration this would mean RAID0
> striping. Common wisdom in Hadoop land has it that a JBOD setup that just
> mounts each disk as a separate directory and does application-level balancing
> over these results in about 30% write-improvement. For example see this claim
> here:
> http://old.nabble.com/Re%3A-RAID-vs.-JBOD-p21466110.html
> It is not clear to me why this would be the case--it seems the RAID
> controller should be able to balance writes as well as the application so it
> may depend on the details of the setup.
> Nonetheless this would be really easy to implement, all you need to do is add
> multiple data directories and balance partition creation over these disks.
> One problem this might cause is if a particular topic is much larger than the
> others it might unbalance the load across the disks. The partition->disk
> assignment policy should probably attempt to evenly spread each topic to
> avoid this, rather than just trying keep the number of partitions balanced
> between disks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
