[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424716#comment-13424716 ] chunhui shen commented on HBASE-6466: - bq.So in startCacheFlush we will now be blocking the incoming writes more right? As more flush threads will try getting the updatesLock.writeLock We will blocking the incoming writes when doing HLog#startCacheFlush now, it is the same as before. Flush threads will try getting the updatesLock.writeLock, it is also same as the before, but it's quite fast, so do you think it's a problem? > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424713#comment-13424713 ] ramkrishna.s.vasudevan commented on HBASE-6466: --- @Chunhui bq.(meaning that while one thread flushes, the others are blocking the inserts and not flushing) The above is one of JD's comments. Currently the patch tries to make more flushes concurrently. So in startCacheFlush we will now be blocking the incoming writes more right? As more flush threads will try getting the updatesLock.writeLock. I may be wrong here. Pls correct me..Thanks.. > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Fix Version/s: 0.96.0
Issue Type: Improvement (was: Bug)
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Fix Version/s: (was: 0.94.1)
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.96.0
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Attachment: HBASE-6459-trunk-v2.patch
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch, HBASE-6459-trunk-v2.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424709#comment-13424709
]
zhou wenjian commented on HBASE-6459:
-
patch for trunk
changed the name to "hbase.hregion.open.and.init.threads.max" as Ted suggests
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Attachment: HBASE-6459-trunk.patch
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch, HBASE-6459-trunk.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424706#comment-13424706 ] Hadoop QA commented on HBASE-6466: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538303/HBASE-6466v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2454//console This message is automatically generated. > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424705#comment-13424705
]
Zhihong Ted Yu commented on HBASE-6459:
---
How about naming the new config param "hbase.hregion.open.and.init.threads.max"
?
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424700#comment-13424700
]
zhou wenjian commented on HBASE-6459:
-
@Ted
the name i used is from the name below in HConstants.java
public static final String HSTORE_OPEN_AND_CLOSE_THREADS_MAX =
"hbase.hstore.open.and.close.threads.max";
would you please give a suitable name?
In your test, did you kill one of the region servers ?
No, I didn't. What i do is to create hregion in parallel (in master), I think
it is not involved with rs.
I'll make a patch for trunk, sorry for that I am not quitely followed the rule
with issue.
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424699#comment-13424699 ] chunhui shen commented on HBASE-6466: - Some performance test data for reference, 1.test environment ： random writting；400 regions per regionserver；row len=50 bytes, value len=1024 bytes;5 regionserver, 300 ipc handler per regionserver;5 client, 50 handler per client for writing 2.test results: before, tps: 7.8k/s per regionserver, Flush:10.1MB/s per regionserver after(2 cache flush handler), tps: 10.7k/s per regionserver, Flush:12.46MB/s per regionserver BTW, under above test, we could increase the tps if increase the client handlers. > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424697#comment-13424697 ] Lars Hofhansl commented on HBASE-6427: -- I see your point. But then you # need a mechanism to only allow a single RegionObserver to implement these hooks. # placed an arbitrary limit on the API (only because it is hard to chain the hooks here, does not mean it is impossible or not useful, as I said one can always create a new implementing class of InternalScanner that does the right thing) This follows the coprocessor API pattern. I think this should be kept. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[ https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424696#comment-13424696 ] Jie Huang commented on HBASE-6464: -- Yes. you are right. It is the issue for 0.94.0 only. thanks. > reportTableInFlux() throws NPE when no table is returned. > - > > Key: HBASE-6464 > URL: https://issues.apache.org/jira/browse/HBASE-6464 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.94.0 >Reporter: Jie Huang >Priority: Critical > Attachments: hbase-6464.patch > > > reportTableInFlux() gets all tables not in flux. However, when no table is > found in getTable(numSkipped) function, it will return null. Then, > errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6466) Enable multi-thread for memstore flush
[ https://issues.apache.org/jira/browse/HBASE-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6466: Attachment: HBASE-6466v3.patch In patch v3, removing lock race in HLog and skipping region who is flushing when flushOneForGlobalPressure > Enable multi-thread for memstore flush > -- > > Key: HBASE-6466 > URL: https://issues.apache.org/jira/browse/HBASE-6466 > Project: HBase > Issue Type: Improvement >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6466.patch, HBASE-6466v2.patch, HBASE-6466v3.patch > > > If the KV is large or Hlog is closed with high-pressure putting, we found > memstore is often above the high water mark and block the putting. > So should we enable multi-thread for Memstore Flush? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[ https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6464: - Affects Version/s: (was: 0.96.0) > reportTableInFlux() throws NPE when no table is returned. > - > > Key: HBASE-6464 > URL: https://issues.apache.org/jira/browse/HBASE-6464 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 0.94.0 >Reporter: Jie Huang >Priority: Critical > Attachments: hbase-6464.patch > > > reportTableInFlux() gets all tables not in flux. However, when no table is > found in getTable(numSkipped) function, it will return null. Then, > errorReporter attempts to print the table number, which causes a NPE here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424690#comment-13424690 ] Zhihong Ted Yu commented on HBASE-6427: --- For preCompactScannerOpen() / preFlushScannerOpen(), they're new APIs and serve particular use case if implemented. I feel there is no need to pass InternalScanner as parameter. Just my personal observation. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6464) reportTableInFlux() throws NPE when no table is returned.
[
https://issues.apache.org/jira/browse/HBASE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424688#comment-13424688
]
ramkrishna.s.vasudevan commented on HBASE-6464:
---
As per the recent code changes I remember a NPE was fixed while getting the
tables.
{code}
HTableDescriptor[] htd = new HTableDescriptor[0];
{code}
I could see this change in getHTableDescriptors().
> reportTableInFlux() throws NPE when no table is returned.
> -
>
> Key: HBASE-6464
> URL: https://issues.apache.org/jira/browse/HBASE-6464
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jie Huang
>Priority: Critical
> Attachments: hbase-6464.patch
>
>
> reportTableInFlux() gets all tables not in flux. However, when no table is
> found in getTable(numSkipped) function, it will return null. Then,
> errorReporter attempts to print the table number, which causes a NPE here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6460) hbck "-repairHoles" shortcut doesn't enable "-fixHdfsOrphans"
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-6460:
--
Summary: hbck "-repairHoles" shortcut doesn't enable "-fixHdfsOrphans"
(was: hbck "-repareHoles" shortcut doesn't enable "-fixHdfsOrphans")
> hbck "-repairHoles" shortcut doesn't enable "-fixHdfsOrphans"
> -
>
> Key: HBASE-6460
> URL: https://issues.apache.org/jira/browse/HBASE-6460
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jie Huang
>Priority: Minor
> Attachments: hbase-6460.patch
>
>
> According to the hbck's help info, shortcut - "-repairHoles" will enable
> "-fixHdfsOrphans" as below.
> {noformat}
> -repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
> -fixHdfsOrphans
> {noformat}
> However, in the implementation, the function "fsck.setFixHdfsOrphans(false);"
> is called in "-repairHoles". This is not consistent with the usage
> information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424684#comment-13424684 ] ramkrishna.s.vasudevan commented on HBASE-6060: --- @Jimmy/@Stack If you can give me a couple of days i can give a patch.. Actually after discussing with Rajesh, the patch 5 has some things to be taken from Stack's patch up in RB. And Stack's patch is not the final one as per the review comments over in RB. > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Reporter: Enis Soztutar >Assignee: rajeshbabu > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, > HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060-trunk_4.patch, > HBASE-6060_trunk_5.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424681#comment-13424681
]
Zhihong Ted Yu commented on HBASE-6459:
---
In your test, did you kill one of the region servers ?
{code}
+ public static final String HREGION_OPEN_AND_INITIALIZE_THREADS_MAX =
"hbase.hregion.open.and.initialize.threads.max";
{code}
The above line is too long (> 100 characters). Can we find a shorter name for
"hbase.hregion.open.and.initialize.threads.max" ?
I would suggest producing patch for trunk so that Hadoop QA can run tests over
it.
After trunk patch passes review, patches for branches can be produced.
Since this is an improvement, I doubt it would go to other branches.
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424680#comment-13424680 ] Lars Hofhansl commented on HBASE-6427: -- Ted, you are trying to redesign the overall coprocessor API. I would prefer that we can keep the API as is, and then work on ways to make better sense of the scanners as part of a different jira. As Andy said a major raison d'etre for coprocessors is to extend HBase. Nobody would just run a 3rd party coprocessor (unless that party is highly trusted). A coprocessor could call System.exit() and shut down the RegionServer... And that was a design choice. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424676#comment-13424676 ] Zhihong Ted Yu commented on HBASE-6427: --- bq. If that is not desired an implementer can break the chain If implementer X breaks the chain, how would implementer Z know that his/her implementation is not broken by someone else ? There is no deterministic order in which X and Z's observers are registered. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424675#comment-13424675
]
zhou wenjian commented on HBASE-6459:
-
we have test the patch 90 in our test cluster (5rs,3u3)
create 2000 regions
before: 80s+
after: in 10s
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424674#comment-13424674 ] Lars Hofhansl commented on HBASE-6427: -- Re: Ted's last comment. So you want to enforce that only a single coprocessor can implement these hooks? I would disagree with that. The chaining is fine (if complicated), no need to mandate what an implementer can and cannot do (see my previous comment). > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424673#comment-13424673 ] Lars Hofhansl commented on HBASE-6427: -- Also another thought: I wonder how hard it is to fold custom timestamps into this. An example of what I mean is time dimension that uses monotonically increasing transaction numbers. A slight API change to the StoreScanner constructor would make it possible to handle that too: By passing in the absolute "time" at which a KV expires rather than the TTL (which it then internally translates to an absolute time anyway, and which in turn depends on the RegionServer's understanding of time, which is not configurable). > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Attachment: HBASE-6459-92.patch
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-92.patch,
> HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424672#comment-13424672 ] Lars Hofhansl commented on HBASE-6427: -- bq. This illustrates the intricacies of scanner chaining. I think that is a false transitive statement. :) For this particular hook chaining is hard, because this is an intricate part of the HBase code. That does not mean that chaining is generally hard (it is not). bq. I wouldn't expect it to function correctly when an arbitrary number of scanners are chained (both) upstream and downstream. Same here... The point is: It is possible to chain region observers even in this case. If that is not desired an implementer can break the chain (via the passed context. and by ignoring the InternalScanner argument). > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Attachment: HBASE-6459-90.patch
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-90.patch, HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424671#comment-13424671
]
Zhihong Ted Yu commented on HBASE-6427:
---
There is no intersection between memstore scanners and store scanners.
The arbitrary number of scanners is due to the following construct:
{code}
+for (RegionEnvironment env: coprocessors) {
+ if (env.getInstance() instanceof RegionObserver) {
+ctx = ObserverContext.createAndPrepare(env, ctx);
+try {
+ s = ((RegionObserver) env.getInstance()).preCompactScannerOpen(ctx,
store, scanners,
+ scanType, earliestPutTs, s);
{code}
I think we should reduce the complexity (due to scanner chaining) for
preCompactScannerOpen().
If we don't provide a working model for scanner chaining, there is no need to
introduce construct for chaining scanners.
> Pluggable compaction policies via coprocessors
> --
>
> Key: HBASE-6427
> URL: https://issues.apache.org/jira/browse/HBASE-6427
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt,
> 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt
>
>
> When implementing higher level stores on top of HBase it is necessary to
> allow dynamic control over how long KVs must be kept around.
> Semi-static config options for ColumnFamilies (# of version or TTL) is not
> sufficient.
> This can be done with a few additional coprocessor hooks, or by makeing
> Store.ScanInfo pluggable.
> Was:
> The simplest way to achieve this is to have a pluggable class to determine
> the smallestReadpoint for Region. That way outside code can control what KVs
> to retain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhou wenjian updated HBASE-6459:
Attachment: HBASE-6459-94.patch
> improve speed of create table
> -
>
> Key: HBASE-6459
> URL: https://issues.apache.org/jira/browse/HBASE-6459
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.0
>Reporter: zhou wenjian
> Fix For: 0.94.1
>
> Attachments: HBASE-6459-94.patch
>
>
> in CreateTableHandler
> for (int regionIdx = 0; regionIdx < this.newRegions.length; regionIdx++) {
> HRegionInfo newRegion = this.newRegions[regionIdx];
> // 1. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> this.fileSystemManager.getRootDir(), this.conf,
> this.hTableDescriptor, null, false, true);
> regionInfos.add(region.getRegionInfo());
> if (regionIdx % batchSize == 0) {
> // 2. Insert into META
> MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
> regionInfos.clear();
> }
> // 3. Close the new region to flush to disk. Close log file too.
> region.close();
> }
> All the region will be create serially.
> If we have thousands of regions, that will be a huge cost.
> We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424670#comment-13424670
]
Jie Huang commented on HBASE-6429:
--
TestFromClientSide is passed on my local server.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hbase.client.TestFromClientSide
Tests run: 56, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 170.132 sec
Results :
Tests run: 56, Failures: 0, Errors: 0, Skipped: 3
{noformat}
> Filter with filterRow() returning true is incompatible with scan with limit
> ---
>
> Key: HBASE-6429
> URL: https://issues.apache.org/jira/browse/HBASE-6429
> Project: HBase
> Issue Type: Bug
> Components: filters
>Affects Versions: 0.96.0
>Reporter: Jason Dai
> Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk-v3.patch,
> hbase-6429-trunk.patch, hbase-6429_0_94_0.patch
>
>
> Currently if we scan with bot limit and a Filter with
> filterRow(List) implemented, an IncompatibleFilterException will
> be thrown. The same exception should also be thrown if the filer has its
> filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424669#comment-13424669 ] Andrew Purtell commented on HBASE-6427: --- bq. I understand the goal for this JIRA and am in support of it. [...] This illustrates the intricacies of scanner chaining. If a scanner is designed for some specific purpose, I wouldn't expect it to function correctly when an arbitrary number of scanners are chained (both) upstream and downstream. Pardon Ted but I think this is an X-Y argument. I'm not sure we are discussing the chaining of _arbitrary_ scanners (unless I have this wrong.) Each CP hook is dealing with a constrained set of scanners. Flushes will be dealing with memstore scanners. Compactions will be dealing with store scanners. The question has been what kind of interface should input parameters and return types share, there's some design give-and-take there. But we are not talking about, for example, combining memstore scanners with store scanners. (At least, I am not.) > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424668#comment-13424668 ] Hadoop QA commented on HBASE-6429: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538293/hbase-6429-trunk-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2453//console This message is automatically generated. > Filter with filterRow() returning true is incompatible with scan with limit > --- > > Key: HBASE-6429 > URL: https://issues.apache.org/jira/browse/HBASE-6429 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk-v3.patch, > hbase-6429-trunk.patch, hbase-6429_0_94_0.patch > > > Currently if we scan with bot limit and a Filter with > filterRow(List) implemented, an IncompatibleFilterException will > be thrown. The same exception should also be thrown if the filer has its > filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424667#comment-13424667 ] Zhihong Ted Yu commented on HBASE-6427: --- I understand the goal for this JIRA and am in support of it. bq. the downstream must do some complicated logic to merge with the previous scanner in ways that we cannot anticipate. This illustrates the intricacies of scanner chaining. If a scanner is designed for some specific purpose, I wouldn't expect it to function correctly when an arbitrary number of scanners are chained (both) upstream and downstream. In fact, chaining introduces unnecessary burden on individual scanner. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424665#comment-13424665 ] Andrew Purtell commented on HBASE-6427: --- And a follow up note to my above comment: It is a goal to provide adequate extension surface (and I include here besides CPs also other pluggable interfaces where performance considerations are paramount) so new features or design alternatives require no core code patches. IMHO, the design strategy for this goal should be incremental, driven by actual use cases, but with foresight added. This issue is a good example of that I think, Lars has really improved flush and compaction hooks, and this work hasn't been done in the abstract. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424664#comment-13424664 ] Andrew Purtell commented on HBASE-6427: --- [~lhofhansl] A follow up JIRA to discuss refactoring the internal scanner interfaces would be reasonable. We do want for extensions to be able to wrap and merge scanners along CP chains. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424663#comment-13424663 ] Andrew Purtell commented on HBASE-6427: --- bq. I personally haven't seen chained scanners in action. If we don't think through how chained scanners work, I wouldn't expect HBase users to use this mechanism. Ted, this is a great comment, because I think it illustrates an incorrect approach to CP API design. (Not meant to be a criticism of you. :-)) CPs are targeted as much for HBase developers as they might be for users. The fundamental goal of CPs is to avoid needing to patch core HBase code for implementing new features or researching design alternatives. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424659#comment-13424659
]
rajeshbabu commented on HBASE-6317:
---
bq. ALREADY_OPENED is not handled by bulk assigned, is that true?
It has handled in trunk as part of HBASE-6012 but not in 94. We can back port
to 94.
But this itself may not solve the problem because
->during clean cluster start up if we set "hbase.master.startup.retainassign"
to true we will call retainAssignment the regions may be assigned to same
region servers before master restart other wise we will get bulk plan by round
robin assignment and we may get different RS than RS before master restart(
which will cause double assignment).
-> in case of failover also while recovering ENABLING tables
if "hbase.master.enabletable.roundrobin" set to true we will get different RS
in bulk plan than RS before restart(double assignment again)
-> lets suppose some of ENABLING table regions are in transition during master
startup they are not added to regionAssignments in regionStates then we will
consider the regions also for assignment in EnableTableHandler which may cause
double assignments.
{code}
if (this.failover == false && this.enablingTables.size() > 0 &&
deadServers.isEmpty()) {
this.failover = true;
}
{code}
bq. Your patch may still not cover all scenarios, for example, if the
deadServers is not empty
failover is false and deadServers not empty means its really clean cluster
startup considering as failover will slow down master startup. If whole cluster
is down then no need to worry about double assignments.
If all the tables are in ENABLING state then failover is false and deadServers
is empty then we can consider as failover and proceed startup.
> Master clean start up and Partially enabled tables make region assignment
> inconsistent.
> ---
>
> Key: HBASE-6317
> URL: https://issues.apache.org/jira/browse/HBASE-6317
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: rajeshbabu
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: HBASE-6317_94.patch, HBASE-6317_94_3.patch
>
>
> If we have a table in partially enabled state (ENABLING) then on HMaster
> restart we treat it as a clean cluster start up and do a bulk assign.
> Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
> leads to region assignment problems. Analysing more on this we found that we
> have better way to handle these scenarios.
> {code}
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
> synchronized (this.regions) {
> regions.put(regionInfo, regionLocation);
> addToServers(regionLocation, regionInfo);
> }
> {code}
> We dont add to regions map so that enable table handler can handle it. But
> as nothing is added to regions map we think it as a clean cluster start up.
> Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424658#comment-13424658 ] Lars Hofhansl commented on HBASE-6427: -- It's used rarely, because coprocessors are not like store procedures, but a method to extend HBase. We *have* through it (IMHO), and the coprocessors now can be chained, and due to the nature of these hooks that will be tricky. For example even if we managed to pass in a KeyValueScanner instance, folks would be tempted to just add this one to the List of scanners down the chain (as was your first thought above), which is tempting, but will not be the right thing to do; the downstream must do some complicated logic to merge with the previous scanner in ways that we cannot anticipate. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424657#comment-13424657 ] Zhihong Ted Yu commented on HBASE-6427: --- bq. to correctly deal with chained scanners I personally haven't seen chained scanners in action. If we don't think through how chained scanners work, I wouldn't expect HBase users to use this mechanism. > Pluggable compaction policies via coprocessors > -- > > Key: HBASE-6427 > URL: https://issues.apache.org/jira/browse/HBASE-6427 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt, > 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt > > > When implementing higher level stores on top of HBase it is necessary to > allow dynamic control over how long KVs must be kept around. > Semi-static config options for ColumnFamilies (# of version or TTL) is not > sufficient. > This can be done with a few additional coprocessor hooks, or by makeing > Store.ScanInfo pluggable. > Was: > The simplest way to achieve this is to have a pluggable class to determine > the smallestReadpoint for Region. That way outside code can control what KVs > to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[ https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6429: - Attachment: hbase-6429-trunk-v3.patch Thanks Ted. Here attaches the updated version. > Filter with filterRow() returning true is incompatible with scan with limit > --- > > Key: HBASE-6429 > URL: https://issues.apache.org/jira/browse/HBASE-6429 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk-v3.patch, > hbase-6429-trunk.patch, hbase-6429_0_94_0.patch > > > Currently if we scan with bot limit and a Filter with > filterRow(List) implemented, an IncompatibleFilterException will > be thrown. The same exception should also be thrown if the filer has its > filterRow() implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424655#comment-13424655
]
Lars Hofhansl commented on HBASE-6427:
--
Thinking on this more... I think the pre{Flush|Compact}ScannerOpen hooks should
just continue to return InternalScanner. Only that interface is needed by
downstream code and we should not extend this only so that coprocessor chaining
becomes simpler.
As it stands these hooks *can* use the passed InternalScanner, but it needs
some understanding of HBase... Which is needed anyway to correctly deal with
chained scanners for flush or compactions.
I.e. I propose leaving it with what the current patch provides. Not opposed to
filing a separate ticket to bring more sense into the various scanner
interfaces we have.
> Pluggable compaction policies via coprocessors
> --
>
> Key: HBASE-6427
> URL: https://issues.apache.org/jira/browse/HBASE-6427
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt,
> 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt
>
>
> When implementing higher level stores on top of HBase it is necessary to
> allow dynamic control over how long KVs must be kept around.
> Semi-static config options for ColumnFamilies (# of version or TTL) is not
> sufficient.
> This can be done with a few additional coprocessor hooks, or by makeing
> Store.ScanInfo pluggable.
> Was:
> The simplest way to achieve this is to have a pluggable class to determine
> the smallestReadpoint for Region. That way outside code can control what KVs
> to retain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424654#comment-13424654 ] Hadoop QA commented on HBASE-5705: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538291/5705-2.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 17 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2452//console This message is automatically generated. > Introduce Protocol Buffer RPC engine > > > Key: HBASE-5705 > URL: https://issues.apache.org/jira/browse/HBASE-5705 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch, > 5705-2.3.patch > > > Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB > aware can be made to go through this RPC engine. The approach, in my current > thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424646#comment-13424646
]
Jimmy Xiang commented on HBASE-6317:
You are right. Another thing, you said ALREADY_OPENED is not handled by bulk
assigned, is that true?
Your patch may still not cover all scenarios, for example, if the deadServers
is not empty:
{noformat}
if (this.failover == false && this.enablingTables.size() > 0 &&
deadServers.isEmpty()) {
this.failover = true;
}
{noformat}
Why not make sure the bulk assigner handle ALREADY_OPENED properly, if not
already?
> Master clean start up and Partially enabled tables make region assignment
> inconsistent.
> ---
>
> Key: HBASE-6317
> URL: https://issues.apache.org/jira/browse/HBASE-6317
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: rajeshbabu
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: HBASE-6317_94.patch, HBASE-6317_94_3.patch
>
>
> If we have a table in partially enabled state (ENABLING) then on HMaster
> restart we treat it as a clean cluster start up and do a bulk assign.
> Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
> leads to region assignment problems. Analysing more on this we found that we
> have better way to handle these scenarios.
> {code}
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
> synchronized (this.regions) {
> regions.put(regionInfo, regionLocation);
> addToServers(regionLocation, regionInfo);
> }
> {code}
> We dont add to regions map so that enable table handler can handle it. But
> as nothing is added to regions map we think it as a clean cluster start up.
> Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5705: --- Attachment: 5705-2.3.patch Noticed that the patch went stale. Updated. (Can this please be committed.) > Introduce Protocol Buffer RPC engine > > > Key: HBASE-5705 > URL: https://issues.apache.org/jira/browse/HBASE-5705 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch, > 5705-2.3.patch > > > Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB > aware can be made to go through this RPC engine. The approach, in my current > thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424622#comment-13424622
]
Zhihong Ted Yu commented on HBASE-6427:
---
Review board doesn't show correct formatting. So I put the following here.
bq. to make a new Interface extending both InternalScanner and KeyValueScanner
I like the above approach.
Currently we have:
{code}
$ find src/main -name '*.java' -exec grep 'ments.*InternalScanner' {} \; -print
* also implements InternalScanner. WARNING: As is, if you try to use this
implements KeyValueScanner, InternalScanner {
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
implements KeyValueScanner, InternalScanner, ChangedReadersObserver {
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
{code}
Combining InternalScanner and KeyValueScanner seems natural.
> Pluggable compaction policies via coprocessors
> --
>
> Key: HBASE-6427
> URL: https://issues.apache.org/jira/browse/HBASE-6427
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt,
> 6427-v3.txt, 6427-v4.txt, 6427-v5.txt, 6427-v7.txt
>
>
> When implementing higher level stores on top of HBase it is necessary to
> allow dynamic control over how long KVs must be kept around.
> Semi-static config options for ColumnFamilies (# of version or TTL) is not
> sufficient.
> This can be done with a few additional coprocessor hooks, or by makeing
> Store.ScanInfo pluggable.
> Was:
> The simplest way to achieve this is to have a pluggable class to determine
> the smallestReadpoint for Region. That way outside code can control what KVs
> to retain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6471: - Fix Version/s: 0.94.2 > Performance regression caused by HBASE-4054 > --- > > Key: HBASE-6471 > URL: https://issues.apache.org/jira/browse/HBASE-6471 > Project: HBase > Issue Type: Bug > Components: client >Affects Versions: 0.92.0 >Reporter: Lars George >Priority: Critical > Fix For: 0.94.2 > > > The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed > to implement HTableInterface. > Since HTable does not have an empty constructor, the patch added a call to > the super() constructor, which though does trigger the ZooKeeper and META > scan, causing a considerable delay. > With multiple threads using the pool in parallel, the first thread is holding > up all the subsequent ones, in effect it negates the whole reason we have a > HTable pool. > We should complete HBASE-5728, or alternatively add a protected, empty > constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424617#comment-13424617 ] Lars Hofhansl commented on HBASE-6471: -- +1 in finishing HBASE-5728... I'm happy to look at that tomorrow. > Performance regression caused by HBASE-4054 > --- > > Key: HBASE-6471 > URL: https://issues.apache.org/jira/browse/HBASE-6471 > Project: HBase > Issue Type: Bug > Components: client >Affects Versions: 0.92.0 >Reporter: Lars George >Priority: Critical > Fix For: 0.94.2 > > > The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed > to implement HTableInterface. > Since HTable does not have an empty constructor, the patch added a call to > the super() constructor, which though does trigger the ZooKeeper and META > scan, causing a considerable delay. > With multiple threads using the pool in parallel, the first thread is holding > up all the subsequent ones, in effect it negates the whole reason we have a > HTable pool. > We should complete HBASE-5728, or alternatively add a protected, empty > constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6471: - Priority: Critical (was: Major) > Performance regression caused by HBASE-4054 > --- > > Key: HBASE-6471 > URL: https://issues.apache.org/jira/browse/HBASE-6471 > Project: HBase > Issue Type: Bug > Components: client >Affects Versions: 0.92.0 >Reporter: Lars George >Priority: Critical > > The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed > to implement HTableInterface. > Since HTable does not have an empty constructor, the patch added a call to > the super() constructor, which though does trigger the ZooKeeper and META > scan, causing a considerable delay. > With multiple threads using the pool in parallel, the first thread is holding > up all the subsequent ones, in effect it negates the whole reason we have a > HTable pool. > We should complete HBASE-5728, or alternatively add a protected, empty > constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424584#comment-13424584
]
rajeshbabu commented on HBASE-6317:
---
@Jimmy,
bq. The inconsistent we are trying to fix here is about the possible
double-assignment, right? Any other inconsistency?
Yes double assignments only. But it may not like calling assign twice for a
region at a time(one region available as online region on two region servers)
bq.If it is just for double-assignment, I think it is better to prevent it from
AssignmentManager (or the new RegionStates introduced in HBASE-6272).
But region state information for all regions may not be available during master
start up. If regions are in transition only we will update region state and put
into rit in processRIT. We may not have region state info for the ENABLING
table regions which are already assigned or assign not yet started.
bq. Whenever a new region is online, if it is assigned to a server different
from that in RegionStates, we can put the corresponding info to a queue
Let's suppose partially enabled regions are going to assign to a RS in the
middle master restarted then the regions may be assigned to other region server
also. We can avoid double assignments happening at a time by region states
introduced in HBASE-6272. But here we may call assign(which may assign region
to RS1) for a region only once after master restart but before restart the
region may be available as online region on other region server(RS2).
Please correct me if i am wrong.
> Master clean start up and Partially enabled tables make region assignment
> inconsistent.
> ---
>
> Key: HBASE-6317
> URL: https://issues.apache.org/jira/browse/HBASE-6317
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: rajeshbabu
> Fix For: 0.92.2, 0.96.0, 0.94.2
>
> Attachments: HBASE-6317_94.patch, HBASE-6317_94_3.patch
>
>
> If we have a table in partially enabled state (ENABLING) then on HMaster
> restart we treat it as a clean cluster start up and do a bulk assign.
> Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
> leads to region assignment problems. Analysing more on this we found that we
> have better way to handle these scenarios.
> {code}
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
> synchronized (this.regions) {
> regions.put(regionInfo, regionLocation);
> addToServers(regionLocation, regionInfo);
> }
> {code}
> We dont add to regions map so that enable table handler can handle it. But
> as nothing is added to regions map we think it as a clean cluster start up.
> Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6429) Filter with filterRow() returning true is incompatible with scan with limit
[
https://issues.apache.org/jira/browse/HBASE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424561#comment-13424561
]
Zhihong Ted Yu commented on HBASE-6429:
---
{code}
+ * This is a Filter wrapper class which is used in the server side. Some filter
+ * related hooks can be defined in this wrapper. The only way to create a
+ * FilterWrapper instance is passing a client side Filter instance.
{code}
To help users make sense of the third sentence, explanation should be added as
to how client Filter instance is obtained by server (through Scan object).
{code}
+ throw new NullPointerException("Cannot create FilterWrapper with null");
{code}
'with null' -> 'with null Filter'
{code}
+//2. Here to merge the filterRow() into filterRow(kvs),
{code}
Rephrase the above: filterRow() is merged with filterRow(kvs)
{code}
-private Filter filter;
+private FilterWrapper filter;
{code}
There is no need to change the type of filter - FilterWrapper doesn't add new
methods.
{code}
+ "Filter hasFilterRow incompatible with scan with limit!");
{code}
Rephrase the above: 'Filter whose hasFilterRow() returns true is '
{code}
+} else {
+ LOG.error("HBaseAdmin is not initialized successfully.");
+}
{code}
The above condition should result in test failure, right ?
{code}
+ } catch (IOException e) {
+e.printStackTrace();
+assertNull("Exception found deleting the table", e);
+ }
{code}
Since exception is part of the assertion, e.printStackTrace() is redundant.
Please remove the call.
{code}
+ * To test if the FilterWrapper keeps the same semantics
{code}
The above sentence seems incomplete.
{code}
+ // no correct result is expected
+ LOG.info("Exception is catched");
+ assertNull("Exception happens in scan", e);
{code}
Spelling mistake in the log statement. Since exception is part of assertion,
the log is redundant. Suggest removing the log statement.
{code}
+ } catch (IOException e) {
+e.printStackTrace();
+assertNull("Exception found deleting the table", e);
+ }
{code}
Please remove e.printStackTrace() call.
> Filter with filterRow() returning true is incompatible with scan with limit
> ---
>
> Key: HBASE-6429
> URL: https://issues.apache.org/jira/browse/HBASE-6429
> Project: HBase
> Issue Type: Bug
> Components: filters
>Affects Versions: 0.96.0
>Reporter: Jason Dai
> Attachments: hbase-6429-trunk-v2.patch, hbase-6429-trunk.patch,
> hbase-6429_0_94_0.patch
>
>
> Currently if we scan with bot limit and a Filter with
> filterRow(List) implemented, an IncompatibleFilterException will
> be thrown. The same exception should also be thrown if the filer has its
> filterRow() implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-6245: - Attachment: hbase-6245.patch It is quite convenient to refresh the page and return that previously selected tab. Here attaches a js solution to keep the previously selected tab while refreshing the page. If it is necessary, we can create a new .js file under hbase-webapps/static/js folder. Any idea, please let me know. > Upon page refresh new UI should return to the previously selected tab > - > > Key: HBASE-6245 > URL: https://issues.apache.org/jira/browse/HBASE-6245 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.96.0 >Reporter: Andrew Purtell >Priority: Minor > Attachments: hbase-6245.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3307) Add checkAndPut to the Thrift API
[ https://issues.apache.org/jira/browse/HBASE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424547#comment-13424547 ] Zhihong Ted Yu commented on HBASE-3307: --- HBASE-1744 introduced thrift2 package. There hasn't been much demand for thrift2 package in 0.92 Can you try out 0.94.1 release ? > Add checkAndPut to the Thrift API > - > > Key: HBASE-3307 > URL: https://issues.apache.org/jira/browse/HBASE-3307 > Project: HBase > Issue Type: New Feature > Components: thrift >Affects Versions: 0.89.20100924 >Reporter: Chris Tarnas >Priority: Minor > > It would be very useful to have the checkAndPut method available via the > Thrift API. This would both allow for easier atomic updates as well as cut > down on at least one Thrift roundtrip for quite a few common tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424546#comment-13424546 ] Zhihong Ted Yu commented on HBASE-6471: --- +1 on completing HBASE-5728. > Performance regression caused by HBASE-4054 > --- > > Key: HBASE-6471 > URL: https://issues.apache.org/jira/browse/HBASE-6471 > Project: HBase > Issue Type: Bug > Components: client >Affects Versions: 0.92.0 >Reporter: Lars George > > The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed > to implement HTableInterface. > Since HTable does not have an empty constructor, the patch added a call to > the super() constructor, which though does trigger the ZooKeeper and META > scan, causing a considerable delay. > With multiple threads using the pool in parallel, the first thread is holding > up all the subsequent ones, in effect it negates the whole reason we have a > HTable pool. > We should complete HBASE-5728, or alternatively add a protected, empty > constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6471) Performance regression caused by HBASE-4054
Lars George created HBASE-6471: -- Summary: Performance regression caused by HBASE-4054 Key: HBASE-6471 URL: https://issues.apache.org/jira/browse/HBASE-6471 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.92.0 Reporter: Lars George The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed to implement HTableInterface. Since HTable does not have an empty constructor, the patch added a call to the super() constructor, which though does trigger the ZooKeeper and META scan, causing a considerable delay. With multiple threads using the pool in parallel, the first thread is holding up all the subsequent ones, in effect it negates the whole reason we have a HTable pool. We should complete HBASE-5728, or alternatively add a protected, empty constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424501#comment-13424501 ] gaojinchao commented on HBASE-4120: --- Hi Liu jia Are you working for this issue now? when do you plan to finish? > isolation and allocation > > > Key: HBASE-4120 > URL: https://issues.apache.org/jira/browse/HBASE-4120 > Project: HBase > Issue Type: New Feature > Components: master, regionserver >Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 >Reporter: Liu Jia >Assignee: Liu Jia > Fix For: 0.96.0 > > Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, > Design_document_for_HBase_isolation_and_allocation_Revised.pdf, > HBase_isolation_and_allocation_user_guide.pdf, > Performance_of_Table_priority.pdf, > Simple_YCSB_Tests_For_TablePriority_Trunk_and_0.90.4.pdf, System > Structure.jpg, TablePriority.patch, TablePriority_v12.patch, > TablePriority_v12.patch, TablePriority_v15_with_coprocessor.patch, > TablePriority_v16_with_coprocessor.patch, TablePriority_v17.patch, > TablePriority_v17.patch, TablePriority_v8.patch, TablePriority_v8.patch, > TablePriority_v8_for_trunk.patch, TablePrioriy_v9.patch > > > The HBase isolation and allocation tool is designed to help users manage > cluster resource among different application and tables. > When we have a large scale of HBase cluster with many applications running on > it, there will be lots of problems. In Taobao there is a cluster for many > departments to test their applications performance, these applications are > based on HBase. With one cluster which has 12 servers, there will be only one > application running exclusively on this server, and many other applications > must wait until the previous test finished. > After we add allocation manage function to the cluster, applications can > share the cluster and run concurrently. Also if the Test Engineer wants to > make sure there is no interference, he/she can move out other tables from > this group. > In groups we use table priority to allocate resource, when system is busy; we > can make sure high-priority tables are not affected lower-priority tables > Different groups can have different region server configurations, some groups > optimized for reading can have large block cache size, and others optimized > for writing can have large memstore size. > Tables and region servers can be moved easily between groups; after changing > the configuration, a group can be restarted alone instead of restarting the > whole cluster. > git entry : https://github.com/ICT-Ope/HBase_allocation . > We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3307) Add checkAndPut to the Thrift API
[ https://issues.apache.org/jira/browse/HBASE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424497#comment-13424497 ] Trung Pham commented on HBASE-3307: --- What is the status of this ticket? Has it been back ported to 0.92? Thanks. > Add checkAndPut to the Thrift API > - > > Key: HBASE-3307 > URL: https://issues.apache.org/jira/browse/HBASE-3307 > Project: HBase > Issue Type: New Feature > Components: thrift >Affects Versions: 0.89.20100924 >Reporter: Chris Tarnas >Priority: Minor > > It would be very useful to have the checkAndPut method available via the > Thrift API. This would both allow for easier atomic updates as well as cut > down on at least one Thrift roundtrip for quite a few common tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
