[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427880#comment-13427880 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539000/HBASE-6372.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient org.apache.hadoop.hbase.regionserver.TestAtomicOperation Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//console This message is automatically generated. > Add scanner batching to Export job > -- > > Key: HBASE-6372 > URL: https://issues.apache.org/jira/browse/HBASE-6372 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 0.96.0, 0.94.2 >Reporter: Lars George >Assignee: Shengsheng Huang >Priority: Minor > Labels: newbie > Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.patch > > > When a single row is too large for the RS heap then an OOME can take out the > entire RS. Setting scanner batching in custom scans helps avoiding this > scenario, but for the supplied Export job this is not set. > Similar to HBASE-3421 we can set the batching to a low number - or if needed > make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4955) Use the official versions of surefire & junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427894#comment-13427894 ] nkeywal commented on HBASE-4955: There is a new surefire 2.12.1 version. It fixes the regression #827, #800 is reopened. Our fix is different, but I suppose that if I didn't propose my fix it was for a good reason. I will need to have a look at this. Will wait a little however to see if someone fixes it. And anyway this new version could contain new regressions, so there is no reasons to be the first users. They're adding stuff into JUnit, so hopefully their will deliver something soon and it won't contain regressions... > Use the official versions of surefire & junit > - > > Key: HBASE-4955 > URL: https://issues.apache.org/jira/browse/HBASE-4955 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.0 > Environment: all >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > > We currently use private versions for Surefire & JUnit since HBASE-4763. > This JIRA traks what we need to move to official versions. > Surefire 2.11 is just out, but, after some tests, it does not contain all > what we need. > JUnit. Could be for JUnit 4.11. Issue to monitor: > https://github.com/KentBeck/junit/issues/359: fixed in our version, no > feedback for an integration on trunk > Surefire: Could be for Surefire 2.12. Issues to monitor are: > 329 (category support): fixed, we use the official implementation from the > trunk > 786 (@Category with forkMode=always): fixed, we use the official > implementation from the trunk > 791 (incorrect elapsed time on test failure): fixed, we use the official > implementation from the trunk > 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on > our version. > 760 (does not take into account the test method): fixed in trunk, not fixed > in our version > 798 (print immediately the test class name): not fixed in trunk, not fixed in > our version > 799 (Allow test parallelization when forkMode=always): not fixed in trunk, > not fixed in our version > 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, > fixed on our version > 800 & 793 are the more important to monitor, it's the only ones that are > fixed in our version but not on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427952#comment-13427952
]
Lars George commented on HBASE-6372:
A few comments on the patch:
{code}
+ final static String EXPORT_BATCHING = "hbase.mapreduce.export.batch";
...
+ " -Dhbase.client.scanner.caching=100\n"
{code}
I see that you followed the config key naming already in use where you declared
your new key. But looking at the other keys being used, they are all over the
place. The one for scanner caching - which is the closest to what you are
adding. I suggest we follow the same rules, i.e. name it
"hbase.mapreduce.export.batch".
{code}
+ try {
+s.setBatch(batching);
+ } catch (Exception e) {
+LOG.error("Batching is not set because : "+e.toString());
+ }
{code}
Why wrap the setBatch() in a try/except? None of the filter being used are of
the kind that trigger the runtime exception. We can add the try/catch later if
ever needed?
{code}
+int batching = conf.getInt(EXPORT_BATCHING,-1);
{code}
Minor not, there should a space between the command the value.
{code}
+LOG.error("Batching is not set because : "+e.toString());
{code}
Same minor nit, no spaces between the string concatenation.
{code}
+System.err.println("For very wide rows consider set scan batching
properties as below:\n"
{code}
Maybe rephrase a bit? For example: "For tables with very wide rows consider
setting the batch size as below:".
> Add scanner batching to Export job
> --
>
> Key: HBASE-6372
> URL: https://issues.apache.org/jira/browse/HBASE-6372
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 0.96.0, 0.94.2
>Reporter: Lars George
>Assignee: Shengsheng Huang
>Priority: Minor
> Labels: newbie
> Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.patch
>
>
> When a single row is too large for the RS heap then an OOME can take out the
> entire RS. Setting scanner batching in custom scans helps avoiding this
> scenario, but for the supplied Export job this is not set.
> Similar to HBASE-3421 we can set the batching to a low number - or if needed
> make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6444) Expose the ability to set custom HTTP Request Headers for the REST client used by RemoteHTable
[ https://issues.apache.org/jira/browse/HBASE-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428043#comment-13428043 ] Erich Hochmuth commented on HBASE-6444: --- You beat me to it :-) Looks like it should work. I'll test it out over the weekend and will report the status. > Expose the ability to set custom HTTP Request Headers for the REST client > used by RemoteHTable > -- > > Key: HBASE-6444 > URL: https://issues.apache.org/jira/browse/HBASE-6444 > Project: HBase > Issue Type: Improvement > Components: rest >Reporter: Erich Hochmuth > Attachments: HBASE-6444-0.94.patch, HBASE-6444.patch, trunk-6444.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > My corporate security office (ISO) requires that all http traffic get routed > through a Web Access Management layer > (http://en.wikipedia.org/wiki/Web_access_management) > Our Hadoop cluster has been segmented by a virtual network with all access to > HBase from outside clients being managed through HBase Stargate rest server. > The corporate WAM system requires that all http clients authenticate with it > first before making any http request to any http service in the corporate > network. After the http client authenticates with the WAM system the WAM > system returns the client a set of values that must be inserted into a http > cookie and request header of all future http requests to other http clients. > This would mean that all requests through the RemoteHTable interface would > require that this cookie and request header be set as part of the http > request. org.apache.hadoop.hbase.rest.client.Client looks like the > appropriate place that this functionality would need to be plugged into. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6359) KeyValue may return incorrect values after readFields()
[ https://issues.apache.org/jira/browse/HBASE-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428103#comment-13428103 ] Zhihong Ted Yu commented on HBASE-6359: --- Integrated to 0.94 and trunk. Thanks for the patch, Dave. Thanks for the review, Lars. > KeyValue may return incorrect values after readFields() > --- > > Key: HBASE-6359 > URL: https://issues.apache.org/jira/browse/HBASE-6359 > Project: HBase > Issue Type: Bug >Reporter: Dave Revell >Assignee: Dave Revell > Labels: noob > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6359-trunk-v1.diff > > > When the same KeyValue object is used multiple times for deserialization > using readFields, some methods may return incorrect values. Here is a > sequence of operations that will reproduce the problem: > # A KeyValue is created whose key has length 10. The private field keyLength > is initialized to 0. > # KeyValue.getKeyLength() is called. This reads the key length 10 from the > backing array and caches it in keyLength. > # KeyValue.readFields() is called to deserialize a new value. The keyLength > field is not cleared and keeps its value of 10, even though this value is > probably incorrect. > # If getKeyLength() is called, the value 10 will be returned. > For example, in a reducer with Iterable, all values after the first > one from the iterable are likely to return incorrect values from > getKeyLength(). > The solution is to clear all memoized values in KeyValue.readFields(). I'll > write a patch for this soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6359) KeyValue may return incorrect values after readFields()
[ https://issues.apache.org/jira/browse/HBASE-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428136#comment-13428136 ] Hudson commented on HBASE-6359: --- Integrated in HBase-TRUNK #3194 (See [https://builds.apache.org/job/HBase-TRUNK/3194/]) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368964) Result = SUCCESS tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java > KeyValue may return incorrect values after readFields() > --- > > Key: HBASE-6359 > URL: https://issues.apache.org/jira/browse/HBASE-6359 > Project: HBase > Issue Type: Bug >Reporter: Dave Revell >Assignee: Dave Revell > Labels: noob > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6359-trunk-v1.diff > > > When the same KeyValue object is used multiple times for deserialization > using readFields, some methods may return incorrect values. Here is a > sequence of operations that will reproduce the problem: > # A KeyValue is created whose key has length 10. The private field keyLength > is initialized to 0. > # KeyValue.getKeyLength() is called. This reads the key length 10 from the > backing array and caches it in keyLength. > # KeyValue.readFields() is called to deserialize a new value. The keyLength > field is not cleared and keeps its value of 10, even though this value is > probably incorrect. > # If getKeyLength() is called, the value 10 will be returned. > For example, in a reducer with Iterable, all values after the first > one from the iterable are likely to return incorrect values from > getKeyLength(). > The solution is to clear all memoized values in KeyValue.readFields(). I'll > write a patch for this soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6359) KeyValue may return incorrect values after readFields()
[ https://issues.apache.org/jira/browse/HBASE-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428146#comment-13428146 ] Hudson commented on HBASE-6359: --- Integrated in HBase-0.94 #381 (See [https://builds.apache.org/job/HBase-0.94/381/]) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java > KeyValue may return incorrect values after readFields() > --- > > Key: HBASE-6359 > URL: https://issues.apache.org/jira/browse/HBASE-6359 > Project: HBase > Issue Type: Bug >Reporter: Dave Revell >Assignee: Dave Revell > Labels: noob > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6359-trunk-v1.diff > > > When the same KeyValue object is used multiple times for deserialization > using readFields, some methods may return incorrect values. Here is a > sequence of operations that will reproduce the problem: > # A KeyValue is created whose key has length 10. The private field keyLength > is initialized to 0. > # KeyValue.getKeyLength() is called. This reads the key length 10 from the > backing array and caches it in keyLength. > # KeyValue.readFields() is called to deserialize a new value. The keyLength > field is not cleared and keeps its value of 10, even though this value is > probably incorrect. > # If getKeyLength() is called, the value 10 will be returned. > For example, in a reducer with Iterable, all values after the first > one from the iterable are likely to return incorrect values from > getKeyLength(). > The solution is to clear all memoized values in KeyValue.readFields(). I'll > write a patch for this soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2315) BookKeeper for write-ahead logging
[ https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428164#comment-13428164 ] Flavio Junqueira commented on HBASE-2315: - Based on your feedback and our own observations from inspecting the code, here is a rough idea of what we would like to do. In the first step, we make HLog an interface exposing the public methods of the current HLog class and make the current HLog class an implementation of the interface. We also create an HLog factory to allow us to instantiate different HLog implementations. Eventually we will have this factory creating BKHLog when we tell it to do so via configuration. So far there is no new functionality. In the second step, we implement BKHLog and decide what to do with the splitter. It is still not entirely clear how to adapt the splitter to BK. Perhaps we don't need a splitter at all with BookKeeper? Let me know if there is any comment about these steps. Otherwise, I'll create two subtasks and start working on the first. > BookKeeper for write-ahead logging > -- > > Key: HBASE-2315 > URL: https://issues.apache.org/jira/browse/HBASE-2315 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Flavio Junqueira > Attachments: HBASE-2315.patch, bookkeeperOverview.pdf, > zookeeper-dev-bookkeeper.jar > > > BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high > throughput write-ahead logging service. This issue provides an implementation > of write-ahead logging for hbase using BookKeeper. Apart from expected > throughput improvements, BookKeeper also has stronger durability guarantees > compared to the implementation currently used by hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2315) BookKeeper for write-ahead logging
[ https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428172#comment-13428172 ] Zhihong Ted Yu commented on HBASE-2315: --- The plan is good. Please use HBASE-5937 for the first step. Thanks > BookKeeper for write-ahead logging > -- > > Key: HBASE-2315 > URL: https://issues.apache.org/jira/browse/HBASE-2315 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Flavio Junqueira > Attachments: HBASE-2315.patch, bookkeeperOverview.pdf, > zookeeper-dev-bookkeeper.jar > > > BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high > throughput write-ahead logging service. This issue provides an implementation > of write-ahead logging for hbase using BookKeeper. Apart from expected > throughput improvements, BookKeeper also has stronger durability guarantees > compared to the implementation currently used by hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6359) KeyValue may return incorrect values after readFields()
[ https://issues.apache.org/jira/browse/HBASE-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6359: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > KeyValue may return incorrect values after readFields() > --- > > Key: HBASE-6359 > URL: https://issues.apache.org/jira/browse/HBASE-6359 > Project: HBase > Issue Type: Bug >Reporter: Dave Revell >Assignee: Dave Revell > Labels: noob > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6359-trunk-v1.diff > > > When the same KeyValue object is used multiple times for deserialization > using readFields, some methods may return incorrect values. Here is a > sequence of operations that will reproduce the problem: > # A KeyValue is created whose key has length 10. The private field keyLength > is initialized to 0. > # KeyValue.getKeyLength() is called. This reads the key length 10 from the > backing array and caches it in keyLength. > # KeyValue.readFields() is called to deserialize a new value. The keyLength > field is not cleared and keeps its value of 10, even though this value is > probably incorrect. > # If getKeyLength() is called, the value 10 will be returned. > For example, in a reducer with Iterable, all values after the first > one from the iterable are likely to return incorrect values from > getKeyLength(). > The solution is to clear all memoized values in KeyValue.readFields(). I'll > write a patch for this soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428181#comment-13428181 ] Flavio Junqueira commented on HBASE-5937: - Just to make sure, there is nothing done related this jira yet, yes? > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Li Pi >Priority: Minor > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6481) SkipFilter javadoc is incorrect
[ https://issues.apache.org/jira/browse/HBASE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6481: -- Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Integrated to trunk. Thanks for the patch, Shrijeet. > SkipFilter javadoc is incorrect > --- > > Key: HBASE-6481 > URL: https://issues.apache.org/jira/browse/HBASE-6481 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Shrijeet Paliwal >Priority: Minor > Labels: newbie > Fix For: 0.96.0 > > Attachments: 0001-HBASE-6481-SkipFilter-javadoc-is-incorrect.patch > > > The javadoc for SkipFilter > (http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SkipFilter.html) > states : > > A wrapper filter that filters an entire row if any of the KeyValue checks do > not pass. > > But the example same javadocs gives to support this statement is wrong. The > *scan.setFilter(new SkipFilter(new ValueFilter(CompareOp.EQUAL, > new BinaryComparator(Bytes.toBytes(0;* , will only emit rows which > have all column values zero. In other words it is going to skip all rows for > which > ValueFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes(0))) does not > pass , which happen to be all non zero valued cells. > In the same example a ValueFilter created with CompareOp.NOT_EQUAL will > filter out the rows which have a column value zero. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428184#comment-13428184 ] Zhihong Ted Yu commented on HBASE-5937: --- Right. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Li Pi >Priority: Minor > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem is problematic
[ https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428217#comment-13428217 ] Dave Revell commented on HBASE-6358: @Todd, can you explain your original rationale for using "srcFs.equals(destFs)" as a way of checking whether the src and dest filesystems are the same? Also, as an HDFS expert, can you suggest the best possible way for checking whether the underlying filesystems are actually the same FS? > Bulkloading from remote filesystem is problematic > - > > Key: HBASE-6358 > URL: https://issues.apache.org/jira/browse/HBASE-6358 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Dave Revell >Assignee: Dave Revell > Attachments: 6358-suggestion.txt, HBASE-6358-trunk-v1.diff, > HBASE-6358-trunk-v2.diff, HBASE-6358-trunk-v3.diff > > > Bulk loading hfiles that don't live on the same filesystem as HBase can cause > problems for subtle reasons. > In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its > own filesystem if it's not already there. Since this can take a long time for > large hfiles, it's likely that the client will timeout and retry. When the > client retries repeatedly, there may be several bulkload operations in flight > for the same hfile, causing lots of unnecessary IO and tying up handler > threads. This can seriously impact performance. In my case, the cluster > became unusable and the regionservers had to be kill -9'ed. > Possible solutions: > # Require that hfiles already be on the same filesystem as HBase in order > for bulkloading to succeed. The copy could be handled by > LoadIncrementalHFiles before the regionserver is called. > # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend > the timeout or something else. > I'm willing to write a patch but I'd appreciate recommendations on how to > proceed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6481) SkipFilter javadoc is incorrect
[ https://issues.apache.org/jira/browse/HBASE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428227#comment-13428227 ] Hudson commented on HBASE-6481: --- Integrated in HBase-TRUNK #3195 (See [https://builds.apache.org/job/HBase-TRUNK/3195/]) HBASE-6481 SkipFilter javadoc is incorrect (Shrijeet) (Revision 1369027) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java > SkipFilter javadoc is incorrect > --- > > Key: HBASE-6481 > URL: https://issues.apache.org/jira/browse/HBASE-6481 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Shrijeet Paliwal >Priority: Minor > Labels: newbie > Fix For: 0.96.0 > > Attachments: 0001-HBASE-6481-SkipFilter-javadoc-is-incorrect.patch > > > The javadoc for SkipFilter > (http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SkipFilter.html) > states : > > A wrapper filter that filters an entire row if any of the KeyValue checks do > not pass. > > But the example same javadocs gives to support this statement is wrong. The > *scan.setFilter(new SkipFilter(new ValueFilter(CompareOp.EQUAL, > new BinaryComparator(Bytes.toBytes(0;* , will only emit rows which > have all column values zero. In other words it is going to skip all rows for > which > ValueFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes(0))) does not > pass , which happen to be all non zero valued cells. > In the same example a ValueFilter created with CompareOp.NOT_EQUAL will > filter out the rows which have a column value zero. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6414) Remove the WritableRpcEngine & associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6414: --- Attachment: 6414-initial.patch.txt Initial patch. Builds. Will start testing it. > Remove the WritableRpcEngine & associated Invocation classes > > > Key: HBASE-6414 > URL: https://issues.apache.org/jira/browse/HBASE-6414 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6414-initial.patch.txt > > > Remove the WritableRpcEngine & Invocation classes once HBASE-5705 gets > committed and all the protocols are rebased to use PB. > Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
Jimmy Xiang created HBASE-6507:
--
Summary: [hbck] TestHBaseFsck ran into TableNotEnabledException
Key: HBASE-6507
URL: https://issues.apache.org/jira/browse/HBASE-6507
Project: HBase
Issue Type: Test
Components: hbck
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Here is one sample exception:
{noformat}
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6507:
---
Attachment: trunk-6507.patch
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6507:
---
Status: Patch Available (was: Open)
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem is problematic
[ https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428291#comment-13428291 ] Todd Lipcon commented on HBASE-6358: hmm... I don't know if I thought about it in a huge amount of detail. The original idea was to allow you to run an MR on one cluster, and then "LoadIncrementalHFiles" on your HBase cluster which uses a different HDFS. I was thinking there would be an advantage here over the distcp-then-load approach, because the region server doing the copy would end up with a local replica after the load. That said, I didn't think through the timeout implications, which seems to be the issue discussed in this JIRA. As for how to determine if they're the same, the .equals() call is supposed to do that, but perhaps it's not working right? > Bulkloading from remote filesystem is problematic > - > > Key: HBASE-6358 > URL: https://issues.apache.org/jira/browse/HBASE-6358 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Dave Revell >Assignee: Dave Revell > Attachments: 6358-suggestion.txt, HBASE-6358-trunk-v1.diff, > HBASE-6358-trunk-v2.diff, HBASE-6358-trunk-v3.diff > > > Bulk loading hfiles that don't live on the same filesystem as HBase can cause > problems for subtle reasons. > In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its > own filesystem if it's not already there. Since this can take a long time for > large hfiles, it's likely that the client will timeout and retry. When the > client retries repeatedly, there may be several bulkload operations in flight > for the same hfile, causing lots of unnecessary IO and tying up handler > threads. This can seriously impact performance. In my case, the cluster > became unusable and the regionservers had to be kill -9'ed. > Possible solutions: > # Require that hfiles already be on the same filesystem as HBase in order > for bulkloading to succeed. The copy could be handled by > LoadIncrementalHFiles before the regionserver is called. > # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend > the timeout or something else. > I'm willing to write a patch but I'd appreciate recommendations on how to > proceed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem is problematic
[ https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428295#comment-13428295 ] Dave Revell commented on HBASE-6358: Thanks Todd. My new plan is to keep the copy, but do it in LoadIncrementalHFiles instead of in Store. This keeps the existing use cases and test cases intact, but works around the timeout/retry problem. > Bulkloading from remote filesystem is problematic > - > > Key: HBASE-6358 > URL: https://issues.apache.org/jira/browse/HBASE-6358 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Dave Revell >Assignee: Dave Revell > Attachments: 6358-suggestion.txt, HBASE-6358-trunk-v1.diff, > HBASE-6358-trunk-v2.diff, HBASE-6358-trunk-v3.diff > > > Bulk loading hfiles that don't live on the same filesystem as HBase can cause > problems for subtle reasons. > In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its > own filesystem if it's not already there. Since this can take a long time for > large hfiles, it's likely that the client will timeout and retry. When the > client retries repeatedly, there may be several bulkload operations in flight > for the same hfile, causing lots of unnecessary IO and tying up handler > threads. This can seriously impact performance. In my case, the cluster > became unusable and the regionservers had to be kill -9'ed. > Possible solutions: > # Require that hfiles already be on the same filesystem as HBase in order > for bulkloading to succeed. The copy could be handled by > LoadIncrementalHFiles before the regionserver is called. > # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend > the timeout or something else. > I'm willing to write a patch but I'd appreciate recommendations on how to > proceed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2315) BookKeeper for write-ahead logging
[ https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428299#comment-13428299 ] Lars Hofhansl commented on HBASE-2315: -- Sounds like fine plan. If you write per region logs you do not need to split at all, but I don't know whether that is feasible to do in BK or not. > BookKeeper for write-ahead logging > -- > > Key: HBASE-2315 > URL: https://issues.apache.org/jira/browse/HBASE-2315 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Flavio Junqueira > Attachments: HBASE-2315.patch, bookkeeperOverview.pdf, > zookeeper-dev-bookkeeper.jar > > > BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high > throughput write-ahead logging service. This issue provides an implementation > of write-ahead logging for hbase using BookKeeper. Apart from expected > throughput improvements, BookKeeper also has stronger durability guarantees > compared to the implementation currently used by hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6456: - Affects Version/s: (was: 0.94.2) I see. I removed 0.94.2 as affected version then. This is probably due to the modularization of HBase in 0.96 then, there will likely be other classes missing. > Export Utility throws ClassNotFoundException > > > Key: HBASE-6456 > URL: https://issues.apache.org/jira/browse/HBASE-6456 > Project: HBase > Issue Type: Bug > Components: mapred, mapreduce, util >Affects Versions: 0.96.0 >Reporter: Shengsheng Huang >Assignee: Shengsheng Huang >Priority: Minor > Attachments: HBASE-6456.patch > > > The command I ran is as below: > "bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1" > And I got the following exceptions: > attempt_201207261322_0002_m_00_0, Status : FAILED > Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:150) > at > org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) > at org.apache.hadoop.mapred.Child$4.run(Child.java:266) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.au > ... > This exception can be resolved by adding hbase common jar into > HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. > We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6476) Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge equivalent
[ https://issues.apache.org/jira/browse/HBASE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428308#comment-13428308 ] Lars Hofhansl commented on HBASE-6476: -- The test is unrelated. I searched the generated findbugs warning for "EnvironmentEdge" and did not find anything. So I think this is good to go. If there are no objections I'll commit this soon. (Then I'll need some help added the build task to check for these, but according to @nkeyval that should be simple). > Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge > equivalent > - > > Key: HBASE-6476 > URL: https://issues.apache.org/jira/browse/HBASE-6476 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6476-v2.txt, 6476-v2.txt, 6476.txt > > > There are still some areas where System.currentTimeMillis() is used in HBase. > In order to make all parts of the code base testable and (potentially) to be > able to configure HBase's notion of time, this should be generally be > replaced with EnvironmentEdgeManager.currentTimeMillis(). > How hard would it be to add a maven task that checks for that, so we do not > introduce System.currentTimeMillis back in the future? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6508) Filter out edits at log split time
Alex Feinberg created HBASE-6508: Summary: Filter out edits at log split time Key: HBASE-6508 URL: https://issues.apache.org/jira/browse/HBASE-6508 Project: HBase Issue Type: Improvement Components: master, regionserver, wal Affects Versions: 0.89-fb Reporter: Alex Feinberg Assignee: Alex Feinberg Fix For: 0.89-fb At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region. This patch does the following: 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map. 2) Send the map to master it as a part of the region server report. 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region. 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time. 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report. This can reduce downtime when a regionserver goes down quite a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428321#comment-13428321
]
Hadoop QA commented on HBASE-6507:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539078/trunk-6507.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 10 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2498//console
This message is automatically generated.
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6505: - Attachment: 6505-v2.txt Updated patch. * performs cleanup when a coprocessor is blacklisted * changed shared map name to "sharedData" * change shareData API to ConcurrentMap (rather than just Map) to set the right expectation for a using CP (i.e. no synchronization needed, etc) Should be good to go. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6505: - Fix Version/s: 0.94.2 0.96.0 Assignee: Lars Hofhansl Status: Patch Available (was: Open) > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428336#comment-13428336
]
Jonathan Hsieh commented on HBASE-6507:
---
This seems like it now has the possibility to hang. Maybe have it timeout with
a test failure after some long period of time? (60s?)
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem is problematic
[ https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428337#comment-13428337 ] Todd Lipcon commented on HBASE-6358: The problem of doing it automatically in LoadIncrementalHFiles (i.e the client) is that it is going to be very slow for any non-trivial amount of data, to funnel it through this single node. Here's an alternate idea: 1. In this JIRA, change the RS side to fail if the filesystem doesn't match 2. Separately, add a new "DistributedLoadIncrementalHFiles" program which acts as a combination of distcp and LoadIncrementalHFiles. For each RS (or perhaps for each region), it would create one map task, with a locality hint to that server. Then the task would copy the relevant file (achieving a local replica) and make the necessary call to load the file. Between step 1 and 2, users would have to use distcp and sacrifice locality. But, with the current scheme, they already don't get locality for the common case where the MR job runs on the same cluster as HBase. Thoughts? > Bulkloading from remote filesystem is problematic > - > > Key: HBASE-6358 > URL: https://issues.apache.org/jira/browse/HBASE-6358 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Dave Revell >Assignee: Dave Revell > Attachments: 6358-suggestion.txt, HBASE-6358-trunk-v1.diff, > HBASE-6358-trunk-v2.diff, HBASE-6358-trunk-v3.diff > > > Bulk loading hfiles that don't live on the same filesystem as HBase can cause > problems for subtle reasons. > In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its > own filesystem if it's not already there. Since this can take a long time for > large hfiles, it's likely that the client will timeout and retry. When the > client retries repeatedly, there may be several bulkload operations in flight > for the same hfile, causing lots of unnecessary IO and tying up handler > threads. This can seriously impact performance. In my case, the cluster > became unusable and the regionservers had to be kill -9'ed. > Possible solutions: > # Require that hfiles already be on the same filesystem as HBase in order > for bulkloading to succeed. The copy could be handled by > LoadIncrementalHFiles before the regionserver is called. > # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend > the timeout or something else. > I'm willing to write a patch but I'd appreciate recommendations on how to > proceed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428350#comment-13428350
]
Jimmy Xiang commented on HBASE-6507:
setupTable pairs with deleteTable. The wait in setupTable should be minimal.
The chance to hang is deleteTable could be higher. I thought about the timeout,
but sure what's a good number. We can start with 60s for small tests, 120s for
big tests, if you agree.
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428350#comment-13428350
]
Jimmy Xiang edited comment on HBASE-6507 at 8/3/12 8:25 PM:
setupTable pairs with deleteTable. The wait in setupTable should be minimal.
The chance to hang in deleteTable could be higher. I thought about the timeout,
but not sure what's a good number. We can start with 60s for small tests, 120s
for big tests, if you agree.
was (Author: jxiang):
setupTable pairs with deleteTable. The wait in setupTable should be
minimal. The chance to hang is deleteTable could be higher. I thought about the
timeout, but sure what's a good number. We can start with 60s for small tests,
120s for big tests, if you agree.
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem is problematic
[ https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428368#comment-13428368 ] Dave Revell commented on HBASE-6358: @Todd, that idea seems fine to me overall. If we just did the slow copy in LoadIncrementalHFiles as I suggested earlier, users would still have the option of doing distcp before calling LoadIncrementalHFiles if they need performance. This has the benefits of # not breaking the current use case of non-local bulk loading when size or speed requirements are modest # not requiring a new DistributedLoadIncrementalHFiles utility This scheme would not give locality though, so users with serious performance requirements might not be satisfied. > Bulkloading from remote filesystem is problematic > - > > Key: HBASE-6358 > URL: https://issues.apache.org/jira/browse/HBASE-6358 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Dave Revell >Assignee: Dave Revell > Attachments: 6358-suggestion.txt, HBASE-6358-trunk-v1.diff, > HBASE-6358-trunk-v2.diff, HBASE-6358-trunk-v3.diff > > > Bulk loading hfiles that don't live on the same filesystem as HBase can cause > problems for subtle reasons. > In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its > own filesystem if it's not already there. Since this can take a long time for > large hfiles, it's likely that the client will timeout and retry. When the > client retries repeatedly, there may be several bulkload operations in flight > for the same hfile, causing lots of unnecessary IO and tying up handler > threads. This can seriously impact performance. In my case, the cluster > became unusable and the regionservers had to be kill -9'ed. > Possible solutions: > # Require that hfiles already be on the same filesystem as HBase in order > for bulkloading to succeed. The copy could be handled by > LoadIncrementalHFiles before the regionserver is called. > # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend > the timeout or something else. > I'm willing to write a patch but I'd appreciate recommendations on how to > proceed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5991) Introduce sequential ZNode based read/write locks
[ https://issues.apache.org/jira/browse/HBASE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428382#comment-13428382 ] Alex Feinberg commented on HBASE-5991: -- (Please disregard last comment, wrong diff linked.) > Introduce sequential ZNode based read/write locks > -- > > Key: HBASE-5991 > URL: https://issues.apache.org/jira/browse/HBASE-5991 > Project: HBase > Issue Type: Improvement >Reporter: Alex Feinberg >Assignee: Alex Feinberg > > This is a continuation of HBASE-5494: > Currently table-level write locks have been implemented using non-sequential > ZNodes as part of HBASE-5494 and committed to 89-fb branch. This issue is to > track converting the table-level locks to sequential ZNodes and supporting > read-write locks, as to solve the issue of preventing schema changes during > region splits or merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428385#comment-13428385 ] Enis Soztutar commented on HBASE-6052: -- I'm mostly done on the patch for this, but one issue came up while working on the migration script. I intend to remove the meta migration code from 0.90 -> 0.92|0.94, and add meta migration from 0.92|0,94 -> 0.96, but if we remove those parts, then 0.90->0.96 won't be supported. The convention for us has been that it is ok to drop support from 0.90 to 0.96, but just wanted to make it explicit that it will indeed be the case. wdyt? > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428396#comment-13428396 ] Zhihong Ted Yu commented on HBASE-6052: --- Direct migration from 0.90 to 0.96 won't be supported. > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428400#comment-13428400 ] Lars Hofhansl commented on HBASE-6052: -- I would agree with that (no path from 0.90-0.96). I do recall discussions on the DEV list suggesting that need to provide such an upgrade path, though. > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428385#comment-13428385 ] Lars Hofhansl edited comment on HBASE-6052 at 8/3/12 9:39 PM: -- I'm mostly done on the patch for this, but one issue came up while working on the migration script. I intend to remove the meta migration code from 0.90 \-> 0.92|0.94, and add meta migration from 0.92|0,94 -> 0.96, but if we remove those parts, then 0.90->0.96 won't be supported. The convention for us has been that it is ok to drop support from 0.90 to 0.96, but just wanted to make it explicit that it will indeed be the case. wdyt? was (Author: enis): I'm mostly done on the patch for this, but one issue came up while working on the migration script. I intend to remove the meta migration code from 0.90 -> 0.92|0.94, and add meta migration from 0.92|0,94 -> 0.96, but if we remove those parts, then 0.90->0.96 won't be supported. The convention for us has been that it is ok to drop support from 0.90 to 0.96, but just wanted to make it explicit that it will indeed be the case. wdyt? > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428385#comment-13428385 ] Lars Hofhansl edited comment on HBASE-6052 at 8/3/12 9:39 PM: -- I'm mostly done on the patch for this, but one issue came up while working on the migration script. I intend to remove the meta migration code from 0.90 \-> 0.92|0.94, and add meta migration from 0.92|0,94 \-> 0.96, but if we remove those parts, then 0.90->0.96 won't be supported. The convention for us has been that it is ok to drop support from 0.90 to 0.96, but just wanted to make it explicit that it will indeed be the case. wdyt? was (Author: enis): I'm mostly done on the patch for this, but one issue came up while working on the migration script. I intend to remove the meta migration code from 0.90 \-> 0.92|0.94, and add meta migration from 0.92|0,94 -> 0.96, but if we remove those parts, then 0.90->0.96 won't be supported. The convention for us has been that it is ok to drop support from 0.90 to 0.96, but just wanted to make it explicit that it will indeed be the case. wdyt? > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6407) Investigate moving to DI (guice) framework for plugin arch.
[ https://issues.apache.org/jira/browse/HBASE-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6407: - Attachment: HBASE-6407-3.patch Lots more work. HMaster should be done. HRegionServer still needs some work. Also I am still trying to figure out the best way to get LocalCluster to play nicely with conf's > Investigate moving to DI (guice) framework for plugin arch. > --- > > Key: HBASE-6407 > URL: https://issues.apache.org/jira/browse/HBASE-6407 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-6407-1.patch, HBASE-6407-2.patch, > HBASE-6407-3.patch > > > Investigate using Guice to inject the correct compat object provided by > compat plugins -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428404#comment-13428404 ] Lars Hofhansl commented on HBASE-6505: -- Looking for some reviewers :) > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428405#comment-13428405 ] Zhihong Ted Yu commented on HBASE-6505: --- Since the sharedData map resides per region, can we introduce some mechanism to limit the amount of heap they consume ? > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
Alex Baranau created HBASE-6509: --- Summary: Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b" Key: HBASE-6509 URL: https://issues.apache.org/jira/browse/HBASE-6509 Project: HBase Issue Type: New Feature Components: filters Reporter: Alex Baranau Assignee: Alex Baranau Priority: Minor Implement fuzzy row key filter to allow fetching records e.g. by this criteria: "???alex?b". This seems to be very useful as an alternative to select records by row keys by specifying their part which is not prefix part. Due to fast-forwarding nature of the filter in many situations this helps to avoid heavy full-table scans. This is especially effective when you have composite row key and (some of) its parts has fixed length. E.g. with the key of format userId_actionId_time, given that userId and actionId length is fixed, one can select user actions of specific type using fuzzy row key by specifying mask "_myaction". Given fast-forwarding nature of filter, this will usually work much faster than doing whole table scan with any of the existing server-side filters. In many cases this can work as secondary-indexing alternative. Many times users implement it as a custom filter and many times they just don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Baranau updated HBASE-6509: Attachment: HBASE-6509.patch Attached patch. Also created review request at https://reviews.apache.org/r/6354/. More unit-tests to come (see review). Please let me know your thoughts about API. I feel like setting mask (0s and 1s only currently) via byte[] is might be overkill. Not sure if we'll need more states for values. Using byte[] just looked handy to me from the client prospective at the time I implemented this initially (some time ago) > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Baranau updated HBASE-6509: Status: Patch Available (was: Open) > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Baranau updated HBASE-6509: Attachment: (was: HBASE-6509.patch) > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Baranau updated HBASE-6509: Attachment: HBASE-6509.patch Sorry - my IDE inserted bad "license header" in patch. Fixed > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6510) Fix HConnection typo in TestFromClientSide
Gregory Chanan created HBASE-6510:
-
Summary: Fix HConnection typo in TestFromClientSide
Key: HBASE-6510
URL: https://issues.apache.org/jira/browse/HBASE-6510
Project: HBase
Issue Type: Bug
Components: test
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Trivial
Fix For: 0.96.0
Attachments: HBASE-6510.patch
{code}
* API that accept a pre-created HConnction instance
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428426#comment-13428426 ] Lars Hofhansl commented on HBASE-6505: -- The sharedData map resides per CP class per RegionServer (not per region). I would not be able to come up with any good limit here, other then saying: "It depends on what the coprocessor is trying to do". It should be up to the coprocessor to be a good citizen (this goes back to the discussion that coprocessors are an HBase extension mechanism). > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6510) Fix HConnection typo in TestFromClientSide
[
https://issues.apache.org/jira/browse/HBASE-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated HBASE-6510:
--
Attachment: HBASE-6510.patch
> Fix HConnection typo in TestFromClientSide
> --
>
> Key: HBASE-6510
> URL: https://issues.apache.org/jira/browse/HBASE-6510
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Trivial
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: HBASE-6510.patch
>
>
> {code}
> * API that accept a pre-created HConnction instance
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6510) Fix HConnection typo in TestFromClientSide
[
https://issues.apache.org/jira/browse/HBASE-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated HBASE-6510:
--
Status: Patch Available (was: Open)
> Fix HConnection typo in TestFromClientSide
> --
>
> Key: HBASE-6510
> URL: https://issues.apache.org/jira/browse/HBASE-6510
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Trivial
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: HBASE-6510.patch
>
>
> {code}
> * API that accept a pre-created HConnction instance
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428432#comment-13428432 ] Hadoop QA commented on HBASE-6509: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539104/HBASE-6509.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestCheckTestClasses Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2500//console This message is automatically generated. > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428443#comment-13428443 ] Hadoop QA commented on HBASE-6505: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539090/6505-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestAssignmentManager org.apache.hadoop.hbase.security.access.TestAccessController Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2499//console This message is automatically generated. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428460#comment-13428460 ] Zhihong Ted Yu commented on HBASE-6509: --- Please categorize TestFuzzyRowFilter as small test. > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6507:
---
Attachment: trunk-6507_v2.patch
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch, trunk-6507_v2.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6507:
---
Status: Open (was: Patch Available)
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch, trunk-6507_v2.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6507:
---
Status: Patch Available (was: Open)
Added time out to the newly added wait loop.
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch, trunk-6507_v2.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6481) SkipFilter javadoc is incorrect
[ https://issues.apache.org/jira/browse/HBASE-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428469#comment-13428469 ] Hudson commented on HBASE-6481: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #120 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/120/]) HBASE-6481 SkipFilter javadoc is incorrect (Shrijeet) (Revision 1369027) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java > SkipFilter javadoc is incorrect > --- > > Key: HBASE-6481 > URL: https://issues.apache.org/jira/browse/HBASE-6481 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: Shrijeet Paliwal >Priority: Minor > Labels: newbie > Fix For: 0.96.0 > > Attachments: 0001-HBASE-6481-SkipFilter-javadoc-is-incorrect.patch > > > The javadoc for SkipFilter > (http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/filter/SkipFilter.html) > states : > > A wrapper filter that filters an entire row if any of the KeyValue checks do > not pass. > > But the example same javadocs gives to support this statement is wrong. The > *scan.setFilter(new SkipFilter(new ValueFilter(CompareOp.EQUAL, > new BinaryComparator(Bytes.toBytes(0;* , will only emit rows which > have all column values zero. In other words it is going to skip all rows for > which > ValueFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes(0))) does not > pass , which happen to be all non zero valued cells. > In the same example a ValueFilter created with CompareOp.NOT_EQUAL will > filter out the rows which have a column value zero. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6359) KeyValue may return incorrect values after readFields()
[ https://issues.apache.org/jira/browse/HBASE-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428468#comment-13428468 ] Hudson commented on HBASE-6359: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #120 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/120/]) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368964) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java > KeyValue may return incorrect values after readFields() > --- > > Key: HBASE-6359 > URL: https://issues.apache.org/jira/browse/HBASE-6359 > Project: HBase > Issue Type: Bug >Reporter: Dave Revell >Assignee: Dave Revell > Labels: noob > Fix For: 0.96.0, 0.94.2 > > Attachments: HBASE-6359-trunk-v1.diff > > > When the same KeyValue object is used multiple times for deserialization > using readFields, some methods may return incorrect values. Here is a > sequence of operations that will reproduce the problem: > # A KeyValue is created whose key has length 10. The private field keyLength > is initialized to 0. > # KeyValue.getKeyLength() is called. This reads the key length 10 from the > backing array and caches it in keyLength. > # KeyValue.readFields() is called to deserialize a new value. The keyLength > field is not cleared and keeps its value of 10, even though this value is > probably incorrect. > # If getKeyLength() is called, the value 10 will be returned. > For example, in a reducer with Iterable, all values after the first > one from the iterable are likely to return incorrect values from > getKeyLength(). > The solution is to clear all memoized values in KeyValue.readFields(). I'll > write a patch for this soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6511) TestAcidGuarantees system test broken by reliance on MiniCluster
Gregory Chanan created HBASE-6511:
-
Summary: TestAcidGuarantees system test broken by reliance on
MiniCluster
Key: HBASE-6511
URL: https://issues.apache.org/jira/browse/HBASE-6511
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.90.7, 0.92.2, 0.96.0, 0.94.1
Reporter: Gregory Chanan
Assignee: Gregory Chanan
When I did HBASE-6334 and HBASE-6379, I only focused on the unit test and
didn't think to test the system test as well.
The problem is that those JIRAs introduced a reliance on the miniCluster:
{code}
// Add a flusher
ctx.addThread(new RepeatingTestThread(ctx) {
public void doAnAction() throws Exception {
util.flush();
}
});
{code}
util.flush requires a miniCluster, so doesn't work with a system test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428475#comment-13428475
]
Jonathan Hsieh commented on HBASE-6507:
---
Please reword fail message to be something like.
"Failed to enable table " + tablename + " after waiting for 60 sec."
After that, +1, feel free to commit.
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch, trunk-6507_v2.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[
https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428494#comment-13428494
]
Gary Helmling commented on HBASE-6505:
--
minor nit: in RegionCoprocessorHost.createEnvironment()
{noformat}
+ if (tmp != null)
+classData = tmp;
{noformat}
should have braces.
Otherwise +1 from me.
I was pondering the constraint of having the shared data map be keyed by String
and whether it would be better to Object keys, but that could lead to attempts
to key data by byte[] with unexpected results, so constraining to Strings might
be good.
> Investigate shared RegionObserver state
> ---
>
> Key: HBASE-6505
> URL: https://issues.apache.org/jira/browse/HBASE-6505
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.96.0, 0.94.2
>
> Attachments: 6505-v2.txt, 6505.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6511) TestAcidGuarantees system test broken by reliance on MiniCluster
[
https://issues.apache.org/jira/browse/HBASE-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428497#comment-13428497
]
Gregory Chanan commented on HBASE-6511:
---
I think I was wrong about this being introduced in HBASE-6334 / HBASE-6379,
removing this links and investigating more.
> TestAcidGuarantees system test broken by reliance on MiniCluster
>
>
> Key: HBASE-6511
> URL: https://issues.apache.org/jira/browse/HBASE-6511
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>
> When I did HBASE-6334 and HBASE-6379, I only focused on the unit test and
> didn't think to test the system test as well.
> The problem is that those JIRAs introduced a reliance on the miniCluster:
> {code}
> // Add a flusher
> ctx.addThread(new RepeatingTestThread(ctx) {
> public void doAnAction() throws Exception {
> util.flush();
> }
> });
> {code}
> util.flush requires a miniCluster, so doesn't work with a system test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6511) TestAcidGuarantees system test broken by reliance on MiniCluster
[
https://issues.apache.org/jira/browse/HBASE-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan resolved HBASE-6511.
---
Resolution: Not A Problem
This is fixed by HBASE-5887 -- was running on an old version.
> TestAcidGuarantees system test broken by reliance on MiniCluster
>
>
> Key: HBASE-6511
> URL: https://issues.apache.org/jira/browse/HBASE-6511
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>
> When I did HBASE-6334 and HBASE-6379, I only focused on the unit test and
> didn't think to test the system test as well.
> The problem is that those JIRAs introduced a reliance on the miniCluster:
> {code}
> // Add a flusher
> ctx.addThread(new RepeatingTestThread(ctx) {
> public void doAnAction() throws Exception {
> util.flush();
> }
> });
> {code}
> util.flush requires a miniCluster, so doesn't work with a system test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6358) Bulkloading from remote filesystem is problematic
[
https://issues.apache.org/jira/browse/HBASE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428508#comment-13428508
]
Todd Lipcon commented on HBASE-6358:
bq. not breaking the current use case of non-local bulk loading when size or
speed requirements are modest
If the size and speed don't matter, then wouldn't you have just used a normal
(non-bulk-load) MR job to load the data?
I think funneling the load through one host basically defeats the purpose of
bulk load. Perhaps it could be available as an option for people just testing
out, but I would prefer the default to be a failure, and you have to enable the
copy with a {{-copyToCluster}} or something.
> Bulkloading from remote filesystem is problematic
> -
>
> Key: HBASE-6358
> URL: https://issues.apache.org/jira/browse/HBASE-6358
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: Dave Revell
>Assignee: Dave Revell
> Attachments: 6358-suggestion.txt, HBASE-6358-trunk-v1.diff,
> HBASE-6358-trunk-v2.diff, HBASE-6358-trunk-v3.diff
>
>
> Bulk loading hfiles that don't live on the same filesystem as HBase can cause
> problems for subtle reasons.
> In Store.bulkLoadHFile(), the regionserver will copy the source hfile to its
> own filesystem if it's not already there. Since this can take a long time for
> large hfiles, it's likely that the client will timeout and retry. When the
> client retries repeatedly, there may be several bulkload operations in flight
> for the same hfile, causing lots of unnecessary IO and tying up handler
> threads. This can seriously impact performance. In my case, the cluster
> became unusable and the regionservers had to be kill -9'ed.
> Possible solutions:
> # Require that hfiles already be on the same filesystem as HBase in order
> for bulkloading to succeed. The copy could be handled by
> LoadIncrementalHFiles before the regionserver is called.
> # Others? I'm not familiar with Hadoop IPC so there may be tricks to extend
> the timeout or something else.
> I'm willing to write a patch but I'd appreciate recommendations on how to
> proceed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Baranau updated HBASE-6509: Attachment: HBASE-6509_1.patch Added SmallTests annotation to test. Thanx, Ted! > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch, HBASE-6509_1.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6052: - Attachment: HBASE-6052_v1.patch Uploaded a patch to https://reviews.apache.org/r/6327/diff/#index_header. Changes HRegionInfo serialization in META and ROOT from Writable to PB. The first version of the patch: - converts references for HRegionInfo serialization into PB - changes most of the code paths that does META operations into using MetaEditor and MetaReader methods. - Support migration from 0.92 and 0.94 to 0.96. Migration code is adapted from the 0.90->0.92 migration code. - Remove classes supporting 0.90 -> 0.92 migration. HRegionInfo090x, MetaMigrationRemovingHTD, etc - Changes ruby shell to use new serialization NOTE: - META table contains info:server and info:startCode as String and int serialized with Bytes.toBytes(). Those are not converted to PB structures. > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6052_v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428529#comment-13428529 ] Andrew Purtell commented on HBASE-6505: --- +1 looks good Lars. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6052: -- Status: Patch Available (was: Open) > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6052_v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428530#comment-13428530 ] Lars Hofhansl commented on HBASE-6505: -- Thanks Gary and Andy. Was thinking about Object keys too, but came to the same conclusion. Will add the braces. :) > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6509) Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by "???alex?b"
[ https://issues.apache.org/jira/browse/HBASE-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428539#comment-13428539 ] Hadoop QA commented on HBASE-6509: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539130/HBASE-6509_1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2501//console This message is automatically generated. > Implement fast-forwarding FuzzyRowFilter to allow filter rows e.g. by > "???alex?b" > - > > Key: HBASE-6509 > URL: https://issues.apache.org/jira/browse/HBASE-6509 > Project: HBase > Issue Type: New Feature > Components: filters >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Attachments: HBASE-6509.patch, HBASE-6509_1.patch > > > Implement fuzzy row key filter to allow fetching records e.g. by this > criteria: "???alex?b". > This seems to be very useful as an alternative to select records by row keys > by specifying their part which is not prefix part. Due to fast-forwarding > nature of the filter in many situations this helps to avoid heavy full-table > scans. > This is especially effective when you have composite row key and (some of) > its parts has fixed length. E.g. with the key of format userId_actionId_time, > given that userId and actionId length is fixed, one can select user actions > of specific type using fuzzy row key by specifying mask "_myaction". > Given fast-forwarding nature of filter, this will usually work much faster > than doing whole table scan with any of the existing server-side filters. > In many cases this can work as secondary-indexing alternative. > Many times users implement it as a custom filter and many times they just > don' know this is possible. Let's add it to the common codebase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6497) Revisit HLog sizing and roll parameters
[ https://issues.apache.org/jira/browse/HBASE-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428544#comment-13428544 ] Harsh J commented on HBASE-6497: bq. Less parallelization during distributed splitting since the unit of distribution is a file. Less parallelization per RS. If you have a lot of RSes, lowering file count does help reduce HBase RPCs too? > Revisit HLog sizing and roll parameters > --- > > Key: HBASE-6497 > URL: https://issues.apache.org/jira/browse/HBASE-6497 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Lars George > > The last major update to the HLog sizing and roll features were done in > HBASE-1394. I am proposing to revisit these settings to overcome recent > issues where the HLog becomes a major bottleneck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428550#comment-13428550 ] Lars Hofhansl commented on HBASE-6505: -- Actually while trying to write a test for this to ensure that the maps for multiple instances of the same CP class are identical, I found that they are in fact not. A new RegionCoprocessorHost is created for each region (see HRegion.), not per RegionServer as I had assumed, defeating the purpose of what I had in mind. Back to the drawing board. The sharedDataMap could be made part of RegionServerServices, instead (and thus reside with the RegionServer). > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6507) [hbck] TestHBaseFsck ran into TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428551#comment-13428551
]
Hadoop QA commented on HBASE-6507:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539118/trunk-6507_v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 10 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2502//console
This message is automatically generated.
> [hbck] TestHBaseFsck ran into TableNotEnabledException
> --
>
> Key: HBASE-6507
> URL: https://issues.apache.org/jira/browse/HBASE-6507
> Project: HBase
> Issue Type: Test
> Components: hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: trunk-6507.patch, trunk-6507_v2.patch
>
>
> Here is one sample exception:
> {noformat}
> org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: tableHDFSRegioininfoMissing
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.(DisableTableHandler.java:75)
> at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6052: - Attachment: HBASE-6052_v2.patch > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6052_v1.patch, HBASE-6052_v2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428554#comment-13428554 ] Andrew Purtell commented on HBASE-6505: --- Yes that is my fault I Should have thought of that. But rather than RegionServerServices how about the CoprocessorHosts sharing a singleton map? > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428556#comment-13428556 ] Lars Hofhansl commented on HBASE-6505: -- NP, and yes a singleton map is better than RegionServerServices. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6052) Convert .META. and -ROOT- content to pb
[ https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428558#comment-13428558 ] Hadoop QA commented on HBASE-6052: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539138/HBASE-6052_v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 67 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2504//console This message is automatically generated. > Convert .META. and -ROOT- content to pb > --- > > Key: HBASE-6052 > URL: https://issues.apache.org/jira/browse/HBASE-6052 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6052_v1.patch, HBASE-6052_v2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428559#comment-13428559 ] Lars Hofhansl commented on HBASE-6505: -- Is it safe to key the sharedDataMap with Class? It's possible that the same class is loaded with different class loaders in different regions, right? So I should probably key that map with a String (which would Class.getName()). Then the next interesting thing: When an Environment is blacklisted, it is only that environment, is it then safe to remove the classes part from the sharedDataMap (since there might be other instance of the same class, but loaded from a different class loader). > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6052) Convert .META. and -ROOT- content to pb
[
https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428560#comment-13428560
]
Zhihong Ted Yu commented on HBASE-6052:
---
The following change to HMaster.java was the cause for patch v2 not to apply:
{code}
- import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
- import org.apache.hadoop.hbase.protobuf.ResponseConverter;
{code}
> Convert .META. and -ROOT- content to pb
> ---
>
> Key: HBASE-6052
> URL: https://issues.apache.org/jira/browse/HBASE-6052
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Enis Soztutar
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: HBASE-6052_v1.patch, HBASE-6052_v2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428562#comment-13428562 ] Andrew Purtell commented on HBASE-6505: --- bq. So I should probably key that map with a String (which would Class.getName()). Makes sense. bq. When an Environment is blacklisted, it is only that environment, is it then safe to remove the classes part from the sharedDataMap (since there might be other instance of the same class, but loaded from a different class loader). Thinking on this, perhaps we can defer that until CPs can actually be unloaded. Blacklisting is only a half measure. If I've seen in an RS log that a CP has been blacklisted my thinking is it's time to cycle the RS. Who knows what damage has been done. And attempts to limit heap usage of the map don't make much sense IMO for the same reason, the CP could leak objects directly. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428562#comment-13428562 ] Andrew Purtell edited comment on HBASE-6505 at 8/4/12 5:24 AM: --- bq. So I should probably key that map with a String (which would Class.getName()). Makes sense. bq. When an Environment is blacklisted, it is only that environment, is it then safe to remove the classes part from the sharedDataMap (since there might be other instance of the same class, but loaded from a different class loader). Thinking on this, perhaps we can defer that until CPs can actually be unloaded. Blacklisting is only a half measure. If I've seen in an RS log that a CP has been blacklisted my thinking is it's time to cycle the RS. Who knows what damage has been done. And attempts to limit heap usage of the map don't make much sense IMO for the same reason, the CP could leak objects directly. Edit: But if you agree, would be good to include a comment to this effect near the map. was (Author: apurtell): bq. So I should probably key that map with a String (which would Class.getName()). Makes sense. bq. When an Environment is blacklisted, it is only that environment, is it then safe to remove the classes part from the sharedDataMap (since there might be other instance of the same class, but loaded from a different class loader). Thinking on this, perhaps we can defer that until CPs can actually be unloaded. Blacklisting is only a half measure. If I've seen in an RS log that a CP has been blacklisted my thinking is it's time to cycle the RS. Who knows what damage has been done. And attempts to limit heap usage of the map don't make much sense IMO for the same reason, the CP could leak objects directly. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6510) Fix HConnection typo in TestFromClientSide
[
https://issues.apache.org/jira/browse/HBASE-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428564#comment-13428564
]
Hadoop QA commented on HBASE-6510:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539106/HBASE-6510.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 10 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2503//console
This message is automatically generated.
> Fix HConnection typo in TestFromClientSide
> --
>
> Key: HBASE-6510
> URL: https://issues.apache.org/jira/browse/HBASE-6510
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Trivial
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: HBASE-6510.patch
>
>
> {code}
> * API that accept a pre-created HConnction instance
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428565#comment-13428565 ] Andrew Purtell commented on HBASE-6505: --- And maybe we'll not get around to making CPs unloadable. I don't want to accidentally put that out there as a design priority. I don't think we have a use case for that. Instead I'd prefer the external coprocessor host route: HBASE-4047. I haven't started implementing due to having other priorities, but that is IMO the way to go for practical and effective resource management in a CP architecture. Unloading is (handwave) a kill -9, with reaping of all resources by the OS. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428569#comment-13428569 ] Lars Hofhansl commented on HBASE-6505: -- Thanks Andy. I do have a new patch that uses a ReferenceMap (from the common collections library) with hard keys and weak values (where the key would be the class name and the value be the class's map). As long as at least one RegionEnvironment exists to hold on to its reference of its map the reference will not be removed from the ReferenceMap (and thus can be shared by other RegionEnvironment for the same class), which is exactly what we want here (and much better than reference counting). When it happens that all Envs for a specific class are blacklisted they will eventually be garbage collected, which then allows the map entry to be collected. Patch with test coming momentarily. If you think that part is overkill and just makes it harder to understand - which might very well be the case, I can easily revert back to the model I had before. > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6505) Investigate shared RegionObserver state
[ https://issues.apache.org/jira/browse/HBASE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6505: - Attachment: 6505-v3.txt New patch: * uses static strong-key/weak-value ReferenceMap as sharedDataMap * uses classnames as keys * has a test the verifies that sharedData maps are in fact identical > Investigate shared RegionObserver state > --- > > Key: HBASE-6505 > URL: https://issues.apache.org/jira/browse/HBASE-6505 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.2 > > Attachments: 6505-v2.txt, 6505-v3.txt, 6505.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
