Thanks a ton for the quick turn around. After one more slow rolling restart
and we will be quite happy.
Cheers,
Randy
On 10/22/15, 10:38 AM, "Vladimir Rodionov" wrote:
>Good to know :)
>
>-Vlad
>
>On Thu, Oct 22, 2015 at 9:40 AM, Randy Fox wrote:
>
>> Hi Vlad,
>>
>> So far patch seems to
Good to know :)
-Vlad
On Thu, Oct 22, 2015 at 9:40 AM, Randy Fox wrote:
> Hi Vlad,
>
> So far patch seems to work perfectly.
>
> -randy
>
>
>
>
> On 10/21/15, 12:52 PM, "Vladimir Rodionov" wrote:
>
> >Randy,
> >
> >You can try patch I just submitted. It is for master but I verified it on
> >1.
Hi Vlad,
So far patch seems to work perfectly.
-randy
On 10/21/15, 12:52 PM, "Vladimir Rodionov" wrote:
>Randy,
>
>You can try patch I just submitted. It is for master but I verified it on
>1.0 branch as well.
>
>-Vlad
>
>On Wed, Oct 21, 2015 at 11:40 AM, Randy Fox wrote:
>
>> https://issu
Randy,
You can try patch I just submitted. It is for master but I verified it on
1.0 branch as well.
-Vlad
On Wed, Oct 21, 2015 at 11:40 AM, Randy Fox wrote:
> https://issues.apache.org/jira/browse/HBASE-14663
>
> -r
>
>
>
> On 10/21/15, 10:35 AM, "Vladimir Rodionov" wrote:
>
> >You are right
https://issues.apache.org/jira/browse/HBASE-14663
-r
On 10/21/15, 10:35 AM, "Vladimir Rodionov" wrote:
>You are right, Randy
>
>This is the bug. Will you open JIRA?
>
>-Vlad
>
>On Wed, Oct 21, 2015 at 9:35 AM, Randy Fox wrote:
>
>> Maybe I am looking in the wrong place but Hstore::close() ha
You are right, Randy
This is the bug. Will you open JIRA?
-Vlad
On Wed, Oct 21, 2015 at 9:35 AM, Randy Fox wrote:
> Maybe I am looking in the wrong place but Hstore::close() has the
> evictOnClose parameter hard coded to true:
>
> // close each store file in parallel
> CompletionService comple
Maybe I am looking in the wrong place but Hstore::close() has the evictOnClose
parameter hard coded to true:
// close each store file in parallel
CompletionService completionService =
new ExecutorCompletionService(storeFileCloserThreadPool);
for (final StoreFile f : result) {
completionServic
I wonder why disabling cache eviction on close does not work in a case of a
bucket cache? I checked the code and did not find
anything suspicious. It has to work.
On Wed, Oct 21, 2015 at 3:52 AM, ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com> wrote:
> Seems that the BucketAllocator#fre
Seems that the BucketAllocator#freeBlock() is synchronized and hence all
the bulk close that it tries to do will be blocked in the synchronized
block. May be something like the IdLock has to be tried here?
Regards
Ram
On Wed, Oct 21, 2015 at 4:20 PM, ramkrishna vasudevan <
ramkrishna.s.vasude...
I think the forceful clearing of the blocks from the bucket cache is
hurting in this case. I think it is worth opening a JIRA for this and work
on a fix.
Regards
Ram
On Wed, Oct 21, 2015 at 12:00 AM, Randy Fox wrote:
> Hi Vlad,
>
> I tried it on a table and on a RegionServer basis and it appea
Hi Vlad,
I tried it on a table and on a RegionServer basis and it appears to have no
affect.
Are we sure it is supported for bucket cache? From my charts the bucket cache
is getting cleared at the same time as the region moves occurred. The regions
slow to move are the ones with bucket cach
Hey, Randy
You can verify your hypothesis by setting hbase.rs.evictblocksonclose to
false for your tables.
-Vlad
On Thu, Oct 15, 2015 at 1:06 PM, Randy Fox wrote:
> Caveat - we are trying to tune the BucketCache (probably a new thread - as
> we are not sure we are getting the most out of it)
>
Caveat - we are trying to tune the BucketCache (probably a new thread - as we
are not sure we are getting the most out of it)
72G off heap
hfile.block.cache.size
0.58
hbase.bucketcache.ioengine
offheap
hbase.bucketcache.size
72800
hbase.bucketcache.bucket.sizes
9
I am a bit curious.
0.94 doesn't have BucketCache.
Can you share BucketCache related config parameters in your cluster ?
Cheers
On Thu, Oct 15, 2015 at 11:11 AM, Randy Fox wrote:
>
> "StoreFileCloserThread-L-1" prio=10 tid=0x027ec800 nid=0xad84
> runnable [0x7fbcc0c65000]
>java
That is not 0.94 vs 1.0 issue - its BucketCache vs LRUCache issue.
It seems that, BucketCache.freeBlock is expensive (and source of
contention). It is the issue. Please open JIRA.
-Vlad
On Thu, Oct 15, 2015 at 11:29 AM, Randy Fox wrote:
> Here is the region server log distilled down to events
Here is the region server log distilled down to events pertinent to the region
move. Not much being logged while the Store is being closed.
This region was about 1.4G compressed, which I consider small.
2015-10-15 08:34:28,510 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Close
dad6
Hi Randy,
As Ted suggested a couple of stack traces would be helpful but if you could
attach the HMaster and a couple of RS logs where you see this problem that
would be great. Also, do you have a large regions that might need some time
to flush? have you looked into any potential problem in HDFS?
Can you give a bit more detail on why block eviction was cause for the slow
region movement?
Did you happen to take stack traces ?
Thanks
> On Oct 15, 2015, at 10:32 AM, Randy Fox wrote:
>
> Hi,
>
> We just upgraded from 0.94 to 1.0.0 and have noticed that region moves are
> super slow (ord
Hi,
We just upgraded from 0.94 to 1.0.0 and have noticed that region moves are
super slow (order of minutes) whereas previously they where in the seconds
range. After looking at the code, I think the time is spent waiting for the
blocks to be evicted from block cache.
I wanted to verify that
19 matches
Mail list logo
