[jira] [Commented] (HBASE-14788) Splitting a region does not support the hbase.rs.evictblocksonclose config when closing source region

2015-11-10 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14999475#comment-14999475
 ] 

Randy Fox commented on HBASE-14788:
---

I tested V3 of the patch and it fixes the split issue.

> Splitting a region does not support the hbase.rs.evictblocksonclose config 
> when closing source region
> -
>
> Key: HBASE-14788
> URL: https://issues.apache.org/jira/browse/HBASE-14788
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.0.0
>Reporter: Randy Fox
>Assignee: Ted Yu
> Attachments: 14788-v1.txt, 14788-v2.txt, 14788-v3.txt
>
>
> i have a table with bucket cache turned on and hbase.rs.evictblocksonclose 
> set to false.  I split a region and watched that the closing of the source 
> region did not complete until the bucketcache was flushed for that region.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14788) Splitting a region does not support the hbase.rs.evictblocksonclose config when closing source region

2015-11-09 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14996885#comment-14996885
 ] 

Randy Fox commented on HBASE-14788:
---

Yes, I grepped the code and there are a few places with closeReader(true), so 
this may not be the last use case for this bug.







> Splitting a region does not support the hbase.rs.evictblocksonclose config 
> when closing source region
> -
>
> Key: HBASE-14788
> URL: https://issues.apache.org/jira/browse/HBASE-14788
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.0.0
>Reporter: Randy Fox
>
> i have a table with bucket cache turned on and hbase.rs.evictblocksonclose 
> set to false.  I split a region and watched that the closing of the source 
> region did not complete until the bucketcache was flushed for that region.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14788) Splitting a region does not support the hbase.rs.evictblocksonclose config when closing source region

2015-11-09 Thread Randy Fox (JIRA) 
Randy Fox created HBASE-14788:
-

 Summary: Splitting a region does not support the 
hbase.rs.evictblocksonclose config when closing source region
 Key: HBASE-14788
 URL: https://issues.apache.org/jira/browse/HBASE-14788
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 1.0.0
Reporter: Randy Fox


i have a table with bucket cache turned on and hbase.rs.evictblocksonclose set 
to false.  I split a region and watched that the closing of the source region 
did not complete until the bucketcache was flushed for that region.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-23 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971587#comment-14971587
 ] 

Randy Fox commented on HBASE-14624:
---

As a baseline (and to install the patch) I did a rolling restart of one RS and 
it took 7 minutes to move the regions.  i waited a few hours for bucket cache 
to fill up and then moved all the regions off and it took 20 seconds or so and 
I verified the buckets freed up.

> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
>  Issue Type: Bug
>  Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt, 14624-v2.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too 
> long to free all the blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-22 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970138#comment-14970138
 ] 

Randy Fox commented on HBASE-14624:
---

yes

> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
>  Issue Type: Bug
>  Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too 
> long to free all the blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-22 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970070#comment-14970070
 ] 

Randy Fox commented on HBASE-14624:
---

The test passed with flying colors.  Regions close super fast with this change.

> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Attachments: 14624-v1.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too 
> long to free all the blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14663) HStore::close does not honor config hbase.rs.evictblocksonclose

2015-10-22 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969425#comment-14969425
 ] 

Randy Fox commented on HBASE-14663:
---

tested in my production cluster and it seems to work fine

> HStore::close does not honor config hbase.rs.evictblocksonclose
> ---
>
> Key: HBASE-14663
> URL: https://issues.apache.org/jira/browse/HBASE-14663
> Project: HBase
>  Issue Type: Bug
>  Components: BlockCache, regionserver
>Affects Versions: 1.0.0
>Reporter: Randy Fox
>Assignee: Vladimir Rodionov
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3
>
> Attachments: HBASE-14663-v1.patch
>
>
> I noticed moving regions was slow and due to the wait for the bucket cache to 
> clear.  I tried setting hbase.rs.evictblocksonclose and it did not help.
> I see the HStore::close method has evictonclose hard coded to true instead of 
> letting the config dictate:
> // close each store file in parallel
> CompletionService completionService =
>new ExecutorCompletionService(storeFileCloserThreadPool);
> for (final StoreFile f : result) {
>completionService.submit(new Callable() {
>  @Override
>  public Void call() throws IOException {
>f.closeReader(true);
>return null;
>  }
>});
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-22 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969419#comment-14969419
 ] 

Randy Fox commented on HBASE-14624:
---

The region that took 3 minutes to close was 1.4G compressed.
The evict blocks on close was not working 
(https://issues.apache.org/jira/browse/HBASE-14663).  The patch for it seems to 
solve the problem.
I can give the above change a try.


> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Attachments: 14624-v1.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too 
> long to free all the blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14663) HStore::close does not honor config hbase.rs.evictblocksonclose

2015-10-21 Thread Randy Fox (JIRA) 
Randy Fox created HBASE-14663:
-

 Summary: HStore::close does not honor config 
hbase.rs.evictblocksonclose
 Key: HBASE-14663
 URL: https://issues.apache.org/jira/browse/HBASE-14663
 Project: HBase
  Issue Type: Bug
  Components: BlockCache, regionserver
Affects Versions: 1.0.0
Reporter: Randy Fox
Priority: Minor


I noticed moving regions was slow and due to the wait for the bucket cache to 
clear.  I tried setting hbase.rs.evictblocksonclose and it did not help.

I see the HStore::close method has evictonclose hard coded to true instead of 
letting the config dictate:

// close each store file in parallel
CompletionService completionService =
   new ExecutorCompletionService(storeFileCloserThreadPool);
for (final StoreFile f : result) {
   completionService.submit(new Callable() {
 @Override
 public Void call() throws IOException {
   f.closeReader(true);
   return null;
 }
   });
}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-19 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963477#comment-14963477
 ] 

Randy Fox commented on HBASE-14624:
---

There are no logs for that. Deduced it from the code and that fact that when i 
added a table to cache it also started taking along time to move when it was 
previously quick.  Vladimir Rodionov suggested i open this Jira after 
discussions on the maillist. 

> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too 
> long to free all the blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-19 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963452#comment-14963452
 ] 

Randy Fox commented on HBASE-14624:
---

The region was about 1.4G compressed and it took about 2.5 minutes to move.  
The bucketcache is configed at 72G, but was not close to full.  The bucket 
sizes: 9216,17408,33792,66560

2015-10-15 08:34:28,510 INFO 
org.apache.hadoop.hbase.regionserver.RSRpcServices: Close 
dad6c71ed395df19220ef1056a110086, moving to 
hb17.prod1.connexity.net,60020,125924021
2015-10-15 08:34:28,510 DEBUG 
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing 
close of 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.
2015-10-15 08:34:28,511 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Closing 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.:
 disabling compactions & flushes
2015-10-15 08:34:28,511 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Running close preflush of 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.
2015-10-15 08:34:28,511 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Started memstore flush for 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.,
 current region memstore size 44.89 MB, and 1/1 column families' memstores are 
being flushed.
2015-10-15 08:34:28,511 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
Couldn't find oldest seqNum for the region we are about to flush: 
[dad6c71ed395df19220ef1056a110086]
2015-10-15 08:34:29,137 INFO 
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed, 
sequenceid=70839856845, memsize=44.9 M, hasBloomFilter=false, into tmp file 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/3600b47839a945de9733cf17581458e0
2015-10-15 08:34:29,144 DEBUG 
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/3600b47839a945de9733cf17581458e0
 as 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/3600b47839a945de9733cf17581458e0
2015-10-15 08:34:29,150 INFO org.apache.hadoop.hbase.regionserver.HStore: Added 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/3600b47839a945de9733cf17581458e0,
 entries=190131, sequenceid=70839856845, filesize=3.2 M
2015-10-15 08:34:29,151 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Finished memstore flush of ~44.89 MB/47066832, currentsize=22.39 KB/22928 for 
region 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.
 in 640ms, sequenceid=70839856845, compaction requested=false
2015-10-15 08:34:29,152 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Updates disabled for region 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.
2015-10-15 08:34:29,152 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Started memstore flush for 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.,
 current region memstore size 22.39 KB, and 1/1 column families' memstores are 
being flushed.
2015-10-15 08:34:29,152 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
Couldn't find oldest seqNum for the region we are about to flush: 
[dad6c71ed395df19220ef1056a110086]
2015-10-15 08:34:29,225 INFO 
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed, 
sequenceid=70839856876, memsize=22.4 K, hasBloomFilter=false, into tmp file 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/a8a74549c7a44174b105cbed23cfaea1
2015-10-15 08:34:29,231 DEBUG 
org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/a8a74549c7a44174b105cbed23cfaea1
 as 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/a8a74549c7a44174b105cbed23cfaea1
2015-10-15 08:34:29,279 INFO org.apache.hadoop.hbase.regionserver.HStore: Added 
hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/a8a74549c7a44174b105cbed23cfaea1,
 entries=128, sequenceid=70839856876, filesize=2.6 K
2015-10-15 08:34:29,280 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
Finished memstore flush of ~22.39 KB/22928, currentsize=0 B/0 for region 
Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.
 in 128ms, sequenceid=70839856876, compaction requested=true

...

2015-10-15 08:37:02,945 INFO org.apache.hadoop.hbase.regionserver.HStore: 
Closed L
2015-10-15 08:37:02,954 DEBUG org.apache.hadoop.hbase.wal.WALSplitter: Wrote 
region 
seqId=hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/recovered.edits/7083985

[jira] [Created] (HBASE-14624) BucketCache.freeBlock is too expensive

2015-10-15 Thread Randy Fox (JIRA) 
Randy Fox created HBASE-14624:
-

 Summary: BucketCache.freeBlock is too expensive
 Key: HBASE-14624
 URL: https://issues.apache.org/jira/browse/HBASE-14624
 Project: HBase
  Issue Type: Improvement
  Components: BlockCache
Affects Versions: 1.0.0
Reporter: Randy Fox


Moving regions is unacceptably slow when using bucket cache, as it takes too 
long to free all the blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-7272) HFileOutputFormat.configureIncrementalLoad should honor table Max File Size and Columns BlockSize

2012-12-17 Thread Randy Fox (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13533999#comment-13533999
 ] 

Randy Fox commented on HBASE-7272:
--

configureIncrementalLoad takes an HTable as a parameter, so shouldn't it be 
able to make these determinations?

> HFileOutputFormat.configureIncrementalLoad should honor table Max File Size 
> and Columns BlockSize
> -
>
> Key: HBASE-7272
> URL: https://issues.apache.org/jira/browse/HBASE-7272
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 0.92.1
>Reporter: Randy Fox
>Priority: Minor
>
> HFileOutputFormat.configureIncrementalLoad is used to generate HFiles 
> matching region assignments.  The problem is that is it does not using file 
> and block size settings.  This can create a lot of files of wrong block size, 
> which may take a long time to compact.  I think it should honor these 
> settings and expedite its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7272) HFileOutputFormat.configureIncrementalLoad should honor tabel Max File Size and Columns BlockSize

2012-12-04 Thread Randy Fox (JIRA) 
Randy Fox created HBASE-7272:


 Summary: HFileOutputFormat.configureIncrementalLoad should honor 
tabel Max File Size and Columns BlockSize
 Key: HBASE-7272
 URL: https://issues.apache.org/jira/browse/HBASE-7272
 Project: HBase
  Issue Type: Improvement
  Components: HFile
Affects Versions: 0.92.1
Reporter: Randy Fox
Priority: Minor


HFileOutputFormat.configureIncrementalLoad is used to generate HFiles matching 
region assignments.  The problem is that is it does not using file and block 
size settings.  This can create a lot of files of wrong block size, which may 
take a long time to compact.  I think it should honor these settings and 
expedite its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira