[jira] [Updated] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7180: - Attachment: 7180-0.94-SKETCH.txt Something nasty like this. There must be a cleaner way. > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 7180-0.94-SKETCH.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7180) RegionScannerImpl.next() is inefficient.
Lars Hofhansl created HBASE-7180: Summary: RegionScannerImpl.next() is inefficient. Key: HBASE-7180 URL: https://issues.apache.org/jira/browse/HBASE-7180 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl We just came across a special scenario. For our Phoenix project (SQL runtime for HBase), we push a lot of work into HBase via coprocessors. One method is to wrap RegionScanner in coprocessor hooks and then do processing in the hook to avoid returning a lot of data to the client unnecessarily. In this specific case this is pretty bad. Since the wrapped RegionScanner's next() does not "know" that it is called this way is still does all of this on each invocation: # Starts a RegionOperation # Increments the request count # set the current read point on a thread local (because generally each call could come from a different thread) # Finally does the next on its StoreScanner(s) # Ends the RegionOperation When this is done in a tight loop millions of times (as is the case for us) it starts to become significant. Not sure what to do about this, really. Opening this issue for discussion. One way is to extend the RegionScanner with an "internal" next() method of sorts, so that all this overhead can be avoided. The coprocessor could call the regular next() methods once and then just call the cheaper internal version. Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Periodically flush the Memstore?
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499603#comment-13499603 ] Lars Hofhansl commented on HBASE-5930: -- I would like to pick this up again and add a flag to Mutation to indicate deferred WAL sync. If HRegion receives a batch of Mutation of which at least one is not marked as deferred the log is sync'ed. Otherwise it is deferred. This will mingle well later with HBASE-5954. > Periodically flush the Memstore? > > > Key: HBASE-5930 > URL: https://issues.apache.org/jira/browse/HBASE-5930 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Priority: Minor > > A colleague of mine ran into an interesting issue. > He inserted some data with the WAL disabled, which happened to fit in the > aggregate Memstores memory. > Two weeks later he a had problem with the HDFS cluster, which caused the > region servers to abort. He found that his data was lost. Looking at the log > we found that the Memstores were not flushed at all during these two weeks. > Should we have an option to flush memstores periodically. There are obvious > downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4047) [Coprocessors] Generic external process host
[ https://issues.apache.org/jira/browse/HBASE-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499577#comment-13499577 ] Andrew Purtell commented on HBASE-4047: --- [~asafm] I didn't get beyond some early high level thoughts. Therefore there is no data, but sure there will be some performance penalty, we must introduce an RPC mechanism between the RegionServer and the child external coprocessor host. It seems reasonable that the external coprocessor host should handle all IPC issues, use Process/ProcessBuilder to launch a child process for hosting the user coprocessor code and get access to its stdin and stdout. We will need to introduce a new type of Observer to the coprocessor framework that can be a singleton watching all regions in the RS. Currently we allocate a coprocessor environment for each region and an Observer can only see what goes on in that environment (for only that region). Otherwise you can imagine for a RS hosting 1000 regions there might be 1000 threads just for IPC between the external coprocessor host in the RS and not one child but 1000. That's a nonstarter. So we want one coprocessor in the RS managing communication to one child, and both parent+child handle all Observer (and Endpoint) actions on all regions, using NIO to multiplex communication among the input and output streams set up by Process/ProcessBuilder. How efficiently this can be done and how low latency it can be kept will determine the performance penalty for external coprocessors. > [Coprocessors] Generic external process host > > > Key: HBASE-4047 > URL: https://issues.apache.org/jira/browse/HBASE-4047 > Project: HBase > Issue Type: New Feature > Components: Coprocessors >Reporter: Andrew Purtell >Assignee: Andrew Purtell > > Where HBase coprocessors deviate substantially from the design (as I > understand it) of Google's BigTable coprocessors is we've reimagined it as a > framework for internal extension. In contrast BigTable coprocessors run as > separate processes colocated with tablet servers. The essential trade off is > between performance, flexibility and possibility, and the ability to control > and enforce resource usage. > Since the initial design of HBase coprocessors some additional considerations > are in play: > - Developing computational frameworks sitting directly on top of HBase hosted > in coprocessor(s); > - Introduction of the map reduce next generation (mrng) resource management > model, and the probability that limits will be enforced via cgroups at the OS > level after this is generally available, e.g. when RHEL 6 deployments are > common; > - The possibility of deployment of HBase onto mrng-enabled Hadoop clusters > via the mrng resource manager and a HBase-specific application controller. > Therefore we should consider developing a coprocessor that is a generic host > for another coprocessor, but one that forks a child process, loads the target > coprocessor into the child, establishes a bidirectional pipe and uses an > eventing model and umbilical protocol to provide for the coprocessor loaded > into the child the same semantics as if it was loaded internally to the > parent, and (eventually) use available resource management capabilities on > the platform -- perhaps via the mrng resource controller or directly with > cgroups -- to limit the child as desired by system administrators or the > application designer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2231) Compaction events should be written to HLog
[ https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2231: - Attachment: 2231-testcase_v2.txt Fixed hang. Test passes now but it should not. We have become lax if a file is missing when we go to compact -- perhaps because region double-assigned -- since we made the CompactionRequest object changes. I think the reasoning is that by the time our compaction runs, another may have run before us so it is ok if the geography is different. Need to fix this first. Let me dig more. > Compaction events should be written to HLog > --- > > Key: HBASE-2231 > URL: https://issues.apache.org/jira/browse/HBASE-2231 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Labels: moved_from_0_20_5 > Fix For: 0.96.0 > > Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt, > 2231v2.txt, hbase-2231-testcase.txt, hbase-2231.txt > > > The sequence for a compaction should look like this: > # Compact region to "new" files > # Write a "Compacted Region" entry to the HLog > # Delete "old" files > This deals with a case where the RS has paused between step 1 and 2 and the > regions have since been reassigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7146) Fix the wrong reference to getReader survived in theFSHLog javadoc
[
https://issues.apache.org/jira/browse/HBASE-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499556#comment-13499556
]
Gustavo Anatoly commented on HBASE-7146:
Hi, Nicolas.
Is the correct reference on javadoc? {code}HLogFactory.createReader{code}
Thanks.
> Fix the wrong reference to getReader survived in theFSHLog javadoc
> --
>
> Key: HBASE-7146
> URL: https://issues.apache.org/jira/browse/HBASE-7146
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Priority: Trivial
>
> In FSHLog, a wrong reference to getReader survived in the javadoc:
> {noformat}
> * To read an HLog, call {@link #getReader(org.apache.hadoop.fs.FileSystem,
> * org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration)}.
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499541#comment-13499541
]
Hadoop QA commented on HBASE-6787:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553923/6787-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3357//console
This message is automatically generated.
> Convert RowProcessorProtocol to protocol buffer service
> ---
>
> Key: HBASE-6787
> URL: https://issues.apache.org/jira/browse/HBASE-6787
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499537#comment-13499537 ] Devaraj Das commented on HBASE-6787: Thanks, Ted for catching that assert (I had forgotten to remove this line from the final patch). I'll upload another patch with the javadoc stuff updated. > Convert RowProcessorProtocol to protocol buffer service > --- > > Key: HBASE-6787 > URL: https://issues.apache.org/jira/browse/HBASE-6787 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499535#comment-13499535
]
Ted Yu commented on HBASE-6787:
---
nit: on review board, I can see a few white spaces.
{code}
+public class RowProcessorClient {
{code}
Please add short javadoc and annotation for audience.
{code}
+ * The request argument contains the processor. The processor object defines
...
* @return The processing result
*/
...
+ public void process(RpcController controller, RowProcessorRequest request,
{code}
There is no return value any more. The request contains information for
constructing processor (see constructRowProcessorFromRequest), I think the
first sentence above should be slightly modified.
{code}
+Class superClass = processor.getClass();
+Type type = superClass.getGenericSuperclass();
{code}
The variable naming above might be a little confusing :-)
{code}
+Type t1 = ((ParameterizedType)type).getActualTypeArguments()[1];
{code}
Do you need to check that there're two type arguments ?
{code}
-public abstract class BaseRowProcessor implements RowProcessor {
+public abstract class BaseRowProcessor
{code}
Better describe what the two Messages mean. This is javadoc at the beginning of
RowProcessor. It would be nice to elaborate above.
> Convert RowProcessorProtocol to protocol buffer service
> ---
>
> Key: HBASE-6787
> URL: https://issues.apache.org/jira/browse/HBASE-6787
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-6787:
--
Attachment: 6787-4.patch
The test failure was caused by a typo:
{code}
assertEquals(0, failures.get());
+assertEquals(0,1);
{code}
Patch v4 removes that hunk.
> Convert RowProcessorProtocol to protocol buffer service
> ---
>
> Key: HBASE-6787
> URL: https://issues.apache.org/jira/browse/HBASE-6787
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499520#comment-13499520
]
Hadoop QA commented on HBASE-6787:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553915/6787-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3356//console
This message is automatically generated.
> Convert RowProcessorProtocol to protocol buffer service
> ---
>
> Key: HBASE-6787
> URL: https://issues.apache.org/jira/browse/HBASE-6787
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6787: --- Attachment: 6787-3.patch Okay here is an updated patch that incorporates all the comments so far. > Convert RowProcessorProtocol to protocol buffer service > --- > > Key: HBASE-6787 > URL: https://issues.apache.org/jira/browse/HBASE-6787 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7179) Region/Threads open/close threads should be daemon threads
Amitanand Aiyer created HBASE-7179: -- Summary: Region/Threads open/close threads should be daemon threads Key: HBASE-7179 URL: https://issues.apache.org/jira/browse/HBASE-7179 Project: HBase Issue Type: Improvement Reporter: Amitanand Aiyer Priority: Minor Fix For: 0.89-fb Can save up to 30 secs during shutdown. No point waiting for these threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6410) Move RegionServer Metrics to metrics2
[ https://issues.apache.org/jira/browse/HBASE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499496#comment-13499496 ] Elliott Clark commented on HBASE-6410: -- Thanks lars. I'll look into it. > Move RegionServer Metrics to metrics2 > - > > Key: HBASE-6410 > URL: https://issues.apache.org/jira/browse/HBASE-6410 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6410-13.patch, HBASE-6410-15.patch, > HBASE-6410-16.patch, HBASE-6410-18.patch, HBASE-6410-1.patch, > HBASE-6410-2.patch, HBASE-6410-3.patch, HBASE-6410-4.patch, > HBASE-6410-5.patch, HBASE-6410-6.patch, HBASE-6410.patch > > > Move RegionServer Metrics to metrics2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7106) [89-fb] Fix the NPE in unit tests for JDK7
[ https://issues.apache.org/jira/browse/HBASE-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499476#comment-13499476 ] Gustavo Anatoly commented on HBASE-7106: Hi, Liyin. About this issue, Jimmy Xiang fixed null qualifier [HBASE-7130|https://issues.apache.org/jira/browse/HBASE-7130], and the jdk7 profile on maven is opened yet to be implemented? > [89-fb] Fix the NPE in unit tests for JDK7 > -- > > Key: HBASE-7106 > URL: https://issues.apache.org/jira/browse/HBASE-7106 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Priority: Trivial > > In JDK7, it will throw out NPE if put a NULL into a TreeSet. And in the unit > tests, user can add a NULL as qualifier into the family map for GET or SCAN. > So we shall do the followings: > 1) Make sure the semantics of NULL column qualifier is equal to that of the > EMPYT_BYTE_ARRAY column qualifier. > 2) An easy fix is to use the EMPYT_BYTE_ARRAY qualifier to replace NULL > qualifier in the family map for the GET or SCAN objects, and everything else > shall be backward compatible. > 3) Add a jdk option in the pom.xml (Assuming user installed the fb packaged > jdk) > eg: mvn test -Dtest=TestFromClientSide -Pjdk7 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6410) Move RegionServer Metrics to metrics2
[ https://issues.apache.org/jira/browse/HBASE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499461#comment-13499461 ] Lars Hofhansl commented on HBASE-6410: -- This left some strange artifact in StoreScanner.java. (Empty if-statements, etc) Looks like the KV size metric was just removed. > Move RegionServer Metrics to metrics2 > - > > Key: HBASE-6410 > URL: https://issues.apache.org/jira/browse/HBASE-6410 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6410-13.patch, HBASE-6410-15.patch, > HBASE-6410-16.patch, HBASE-6410-18.patch, HBASE-6410-1.patch, > HBASE-6410-2.patch, HBASE-6410-3.patch, HBASE-6410-4.patch, > HBASE-6410-5.patch, HBASE-6410-6.patch, HBASE-6410.patch > > > Move RegionServer Metrics to metrics2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7026) Make metrics collection in StoreScanner.java more efficient
[
https://issues.apache.org/jira/browse/HBASE-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499459#comment-13499459
]
Lars Hofhansl commented on HBASE-7026:
--
Indeed it's from HBASE-6410
> Make metrics collection in StoreScanner.java more efficient
> ---
>
> Key: HBASE-7026
> URL: https://issues.apache.org/jira/browse/HBASE-7026
> Project: HBase
> Issue Type: Sub-task
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
>
> Per the benchmarks I ran, the following block of code seems to be inefficient:
> StoreScanner.java:
> public synchronized boolean next(List outResult, int limit,
> String metric) throws IOException {
> // ...
> // update the counter
> if (addedResultsSize > 0 && metric != null) {
> HRegion.incrNumericMetric(this.metricNamePrefix + metric,
> addedResultsSize);
> }
> // ...
> Removing this block increased throughput by 10%. We should move this to the
> outer layer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7026) Make metrics collection in StoreScanner.java more efficient
[
https://issues.apache.org/jira/browse/HBASE-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499458#comment-13499458
]
Lars Hofhansl commented on HBASE-7026:
--
Looks like in trunk this is already gone?!
I see stuff like this:
{code}
if (cumulativeMetric > 0 && metric != null) {
}
{code}
An empty if statement, that does not look like it was on purpose.
[~eclark] Is this by accident from the metric refactor?
> Make metrics collection in StoreScanner.java more efficient
> ---
>
> Key: HBASE-7026
> URL: https://issues.apache.org/jira/browse/HBASE-7026
> Project: HBase
> Issue Type: Sub-task
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
>
> Per the benchmarks I ran, the following block of code seems to be inefficient:
> StoreScanner.java:
> public synchronized boolean next(List outResult, int limit,
> String metric) throws IOException {
> // ...
> // update the counter
> if (addedResultsSize > 0 && metric != null) {
> HRegion.incrNumericMetric(this.metricNamePrefix + metric,
> addedResultsSize);
> }
> // ...
> Removing this block increased throughput by 10%. We should move this to the
> outer layer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7026) Make metrics collection in StoreScanner.java more efficient
[
https://issues.apache.org/jira/browse/HBASE-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499456#comment-13499456
]
Lars Hofhansl commented on HBASE-7026:
--
Yeah, we already removed this from the inner loop (dealing with the case of
wide columns).
But for tall, narrow tables this is still a problem.
I'd be happy to just dump that metric, what is it good for anyway?
> Make metrics collection in StoreScanner.java more efficient
> ---
>
> Key: HBASE-7026
> URL: https://issues.apache.org/jira/browse/HBASE-7026
> Project: HBase
> Issue Type: Sub-task
>Reporter: Karthik Ranganathan
>Assignee: Karthik Ranganathan
>
> Per the benchmarks I ran, the following block of code seems to be inefficient:
> StoreScanner.java:
> public synchronized boolean next(List outResult, int limit,
> String metric) throws IOException {
> // ...
> // update the counter
> if (addedResultsSize > 0 && metric != null) {
> HRegion.incrNumericMetric(this.metricNamePrefix + metric,
> addedResultsSize);
> }
> // ...
> Removing this block increased throughput by 10%. We should move this to the
> outer layer.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4047) [Coprocessors] Generic external process host
[ https://issues.apache.org/jira/browse/HBASE-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499401#comment-13499401 ] Asaf Mesika commented on HBASE-4047: This truly sounds like a great feature. I was wondering: * Did you find any performance penalties for shifting data back and forth between the processes? Did you this using the loopback interface? * What method did you choose to communicate between those processes? TCP? Output stream piping? > [Coprocessors] Generic external process host > > > Key: HBASE-4047 > URL: https://issues.apache.org/jira/browse/HBASE-4047 > Project: HBase > Issue Type: New Feature > Components: Coprocessors >Reporter: Andrew Purtell >Assignee: Andrew Purtell > > Where HBase coprocessors deviate substantially from the design (as I > understand it) of Google's BigTable coprocessors is we've reimagined it as a > framework for internal extension. In contrast BigTable coprocessors run as > separate processes colocated with tablet servers. The essential trade off is > between performance, flexibility and possibility, and the ability to control > and enforce resource usage. > Since the initial design of HBase coprocessors some additional considerations > are in play: > - Developing computational frameworks sitting directly on top of HBase hosted > in coprocessor(s); > - Introduction of the map reduce next generation (mrng) resource management > model, and the probability that limits will be enforced via cgroups at the OS > level after this is generally available, e.g. when RHEL 6 deployments are > common; > - The possibility of deployment of HBase onto mrng-enabled Hadoop clusters > via the mrng resource manager and a HBase-specific application controller. > Therefore we should consider developing a coprocessor that is a generic host > for another coprocessor, but one that forks a child process, loads the target > coprocessor into the child, establishes a bidirectional pipe and uses an > eventing model and umbilical protocol to provide for the coprocessor loaded > into the child the same semantics as if it was loaded internally to the > parent, and (eventually) use available resource management capabilities on > the platform -- perhaps via the mrng resource controller or directly with > cgroups -- to limit the child as desired by system administrators or the > application designer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6830) [WINDOWS] Tests should not rely on local temp dir to be available in DFS
[
https://issues.apache.org/jira/browse/HBASE-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499394#comment-13499394
]
Hudson commented on HBASE-6830:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #266 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/266/])
HBASE-6830. [WINDOWS] Tests should not rely on local temp dir to be
available in DFS (Revision 1410659)
Result = SUCCESS
enis :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
> [WINDOWS] Tests should not rely on local temp dir to be available in DFS
>
>
> Key: HBASE-6830
> URL: https://issues.apache.org/jira/browse/HBASE-6830
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Labels: window
> Fix For: 0.96.0
>
> Attachments: hbase-6830_v1-0.94.patch, hbase-6830_v1-trunk.patch,
> hbase-6830_v5-0.94.patch, hbase-6830_v5-trunk.patch
>
>
> Some of the tests resolve the local temp directory for temporary test data,
> but use this directory path in dfs. Since on windows, local temp dir is
> resolved to something like: c:\\,
> DistributedFileSystem.getPathName() throws an IllegalArgumentException
> complaining that it is not a valid path name.
> Instead of relying on a local temp dir name, we should create a temp dir on
> dfs, and use this as a basis dir for test data.
> At least the following test cases are affected by this:
> {code}
> TestHFileOutputFormat
> TestHRegionServerBulkLoad
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2645) HLog writer can do 1-2 sync operations after lease has been recovered for split process.
[ https://issues.apache.org/jira/browse/HBASE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499395#comment-13499395 ] Hudson commented on HBASE-2645: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #266 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/266/]) HBASE-2645 HLog writer can do 1-2 sync operations after lease has been recovered for split process (Revision 1410651) HBASE-2645 HLog writer can do 1-2 sync operations after lease has been recovered for split process (Revision 1410631) Result = SUCCESS stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java > HLog writer can do 1-2 sync operations after lease has been recovered for > split process. > > > Key: HBASE-2645 > URL: https://issues.apache.org/jira/browse/HBASE-2645 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 0.90.4 >Reporter: Cosmin Lehene >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 2645-addendum.txt, 2645_hacking.txt, 2645.txt, > 2645v2.txt, 2645v3.txt, 2645v4.txt, hdfs_1.0_editswriter_recoverlease.txt, > hdfs_trunk_editswriter_recoverlease.txt, > org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit-output.txt, > org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit-output.txt > > > TestHLogSplit.testLogCannotBeWrittenOnceParsed is failing. > This test starts a thread that writes one edit to the log, syncs and counts. > During this, a HLog.splitLog operation is started. splitLog recovers the log > lease before reading the log, so that the original regionserver could not > wake up and write after the split process started. > The test compares the number of edits reported by the split process and by > the writer thread. Writer thread (called zombie in the test) should report <= > than the splitLog (sync() might raise after the last edit gets written and > the edit won't get counted by zombie thread). However it appears that the > zombie counts 1-2 more edits. So it looks like it can sync without a lease. > This might be a hdfs-0.20 related issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
