[
https://issues.apache.org/jira/browse/BLUR-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442942#comment-13442942
]
Scott Leberknight commented on BLUR-15:
---------------------------------------
The BlockCache has three constructors, two of which simply call other
constructors; one of them does hardcode 128MB as the slab size, but the other
of the two takes a slabSize parameter and hardcodes the block size as 32MB. The
third constructor accepts arguments for the totalMemory, slabSize, and
blockSize, and then computes the number of blocks per slab and the number of
slabs. The first two constructors are never called by any other code so far as
I can tell. The third constructor (the one that takes totalMemory, slabSize,
and blockSize) is only called by ThriftBlurShardServer's static createServer
method.
So, the BlockCache actually does permit construction using any slab size you
like. The ThriftBlurShardServer has three inputs in the static createServer
method: a hardcoded numberOfBlocksPerSlab value of 16,384; a hardcoded
blockSize of 8,192 (from the constant BlockDirectory.BLOCK_SIZE); and the
slabCount from the "blur.shard.blockcache.slab.count" configuration parameter.
From those three inputs it then computes slabSize and totalMemory, and then
instantiates a BlockCache using those two inputs plus the blockSize, and then
BlockCache turns around and re-computes numberOfBlockPerSlab and numberOfSlabs
in its constructor.
So it seems a little roundabout the way all these sizes are calculated and
passed into the BlockCache via the ThriftBlurShardServer, but you can in fact
instantiate a BlockCache with whatever slabSize you want. So the question is
how this should be changed, if at all. It seems the same logic is basically in
two places, but for different reasons, i.e. because different values are
calculated. It also seems like certain restrictions/invariants should be
enforced, e.g. that (slabSize % blockSize) = 0 and (totalMemory % slabSize) =
0. These probably should be enforced in BlockCache, which should contain all
this logic. Though I see why ThriftBlurShardServer has a hand in it, which is
to catch an OutOfMemoryError in case the configuration results in that error.
You could basically have three configuration parameters that
ThriftBlurShardServer would use:
1. the slab count (this already is a config parameter)
2. the block size
3. the number of blocks per slab
With these three the total memory and slab size can be controlled, and passed
to BlockCache without it being modified (other than to enforce the invariants).
Does this make sense?
> Change the Block Cache to allow for a configurable slab size
> ------------------------------------------------------------
>
> Key: BLUR-15
> URL: https://issues.apache.org/jira/browse/BLUR-15
> Project: Apache Blur
> Issue Type: Improvement
> Reporter: Aaron McCurry
>
> The current implementation of the block cache has a hard coded slab size of
> 128MB. This should be configurable so that unit testing can use the block
> cache at a smaller size.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
