[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Affects Version/s: 0.16.0 Fix Version/s: (was: 0.16.0) 0.17.0 Putting this off until later, as the performance hit is pretty substantial, and I think there might be a solution that gives us both performance and correctness. [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Affects Versions: 0.16.0 Reporter: Bryan Duxbury Assignee: stack Fix For: 0.17.0 Attachments: 2512-v2.patch, 2513.patch Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2667) [hbase] Clients should be able to open an OutputStream for putting a value into a table
[ https://issues.apache.org/jira/browse/HADOOP-2667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12560731#action_12560731 ] Bryan Duxbury commented on HADOOP-2667: --- I'm not sure that streaming multi-gigabyte blobs into HBase really makes any sense. If you're going to have a lot of multi-gigabyte blobs, HDFS is probably a better fit for you, and just store the metadata in HBase. However, I guess I could see situations where you might want to stream something like a few megabytes rather than keep it all in memory. However, as I understand it, the current RPC system probably wouldn't support this very well, so it would bear some scrutiny. [hbase] Clients should be able to open an OutputStream for putting a value into a table --- Key: HADOOP-2667 URL: https://issues.apache.org/jira/browse/HADOOP-2667 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Priority: Minor Right now the only interface for insertion is via a byte[], which severely limits the flexibility of the system. Added 2007-10-08 by stuhood. Tell us more Stu why you need this feature? (Because you want to write BLOBs of multiple Gigabytes into an hbase cell?) Added as part of migrating new feature requests from the obsoleted http://wiki.apache.org/hadoop/Hbase/HbaseFeatureRequests -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2666) [hbase] Cleanup: All generated code should go into the o.a.h.h.generated package
[ https://issues.apache.org/jira/browse/HADOOP-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12560733#action_12560733 ] Bryan Duxbury commented on HADOOP-2666: --- -1 on this one. It seems like it would be a little confusing. If there's a thrift package, then the generated code for thrift should probably be under it. [hbase] Cleanup: All generated code should go into the o.a.h.h.generated package Key: HADOOP-2666 URL: https://issues.apache.org/jira/browse/HADOOP-2666 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Priority: Trivial As is, generated code goes into o.a.h.h.thrift.generated, o.a.h.h.hql.generated, etc. Rather, have it all go into the one package, o.a.h.h.generated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12560734#action_12560734 ] Bryan Duxbury commented on HADOOP-2513: --- I understand that the performance degrades as a result of this patch, but I'm much more worried about the correctness of the data than the performance. Sure, it might make it faster to leave this out, but the first time someone happens to hit this problem, they'll go out of their minds trying to track it down. I think we can make some improvements that will make this patch a little faster. I will take another look and try to freshen up the patch. [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: stack Fix For: 0.16.0 Attachments: 2512-v2.patch, 2513.patch Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2637) [hbase] WrongRegionException looks suspiciously like client region selection is doing its calc. wrong
[
https://issues.apache.org/jira/browse/HADOOP-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559975#action_12559975
]
Bryan Duxbury commented on HADOOP-2637:
---
Just so I know I'm reading the exception correctly, this indicates that it's
trying to operate on a region that spans from 103481 to 163328, and the
row we're trying to commit on is actually the end key, right?
Well, looking over the location caching code, it can't be that, because it
specifically tests for endKey. That means that you had to get this wrong
region directly from the .META. region. The methods for doing that don't check
to see if the key you are trying to locate is endKey of the regioninfo we
bring back. We could add this check, but I'm not sure it would be much help.
We're using getClosestRowBefore to figure out what key in the .META. to query
for region info, and if it's coming back with the region immediately before it,
the only thing I could think that would be happening is it's in the middle of a
write or something and hasn't written the new regioninfo row yet. Otherwise, it
should come up.
Did you see lots of these errors, or just one? Not that it's excusable at any
rate, but if it's pretty frequent, it'd mean there was a much larger problem
than if this was incidental.
[hbase] WrongRegionException looks suspiciously like client region selection
is doing its calc. wrong
-
Key: HADOOP-2637
URL: https://issues.apache.org/jira/browse/HADOOP-2637
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Reporter: stack
Priority: Minor
I got this this evening:
{code}
08/01/17 05:57:06 ERROR hbase.PerformanceEvaluation: Failed
org.apache.hadoop.hbase.WrongRegionException:
org.apache.hadoop.hbase.WrongRegionException: Requested row out of range for
HRegion TestTable,103481,1200548919843, startKey='103481',
getEndKey()='163328', row='163328'
at org.apache.hadoop.hbase.HRegion.checkRow(HRegion.java:1491)
at org.apache.hadoop.hbase.HRegion.obtainRowLock(HRegion.java:1536)
at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1231)
at
org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1497)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:968)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:939)
at
org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:461)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:329)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:525)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runNIsOne(PerformanceEvaluation.java:546)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:568)
at
org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:663)
at
org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:682)
..
{code}
We seem to be asking for the endkey on a region rather than asking for the
next region, the one that has the asked-for row as its start key.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Component/s: contrib/hbase [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table Key: HADOOP-2631 URL: https://issues.apache.org/jira/browse/HADOOP-2631 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Clint Morgan noticed that the current HTable#getStartKeys implementation isn't correct when there are more than one table, or the table you're trying to get the start keys of isn't the only table. I believe that the solution is making sure that the scanner starts at the key that should lead off your table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Attachment: 2631.patch This patch seems to resolve the problem. It still breaks out of the scanner loop when the table name doesn't match the table we're searching for, but it also starts the scanner at the name of the table we're searching for instead of EMPTY_START_ROW, which was causing rows for different tables to pop up first and prematurely end the scan. [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table Key: HADOOP-2631 URL: https://issues.apache.org/jira/browse/HADOOP-2631 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Attachments: 2631.patch Clint Morgan noticed that the current HTable#getStartKeys implementation isn't correct when there are more than one table, or the table you're trying to get the start keys of isn't the only table. I believe that the solution is making sure that the scanner starts at the key that should lead off your table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Attachment: 2631-v2.patch New TestGetStartKeys was unreliable in last patch. This one passes consistently. [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table Key: HADOOP-2631 URL: https://issues.apache.org/jira/browse/HADOOP-2631 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Attachments: 2631-v2.patch, 2631.patch Clint Morgan noticed that the current HTable#getStartKeys implementation isn't correct when there are more than one table, or the table you're trying to get the start keys of isn't the only table. I believe that the solution is making sure that the scanner starts at the key that should lead off your table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Status: Patch Available (was: Open) Trying Hudson. [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table Key: HADOOP-2631 URL: https://issues.apache.org/jira/browse/HADOOP-2631 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Attachments: 2631-v2.patch, 2631.patch Clint Morgan noticed that the current HTable#getStartKeys implementation isn't correct when there are more than one table, or the table you're trying to get the start keys of isn't the only table. I believe that the solution is making sure that the scanner starts at the key that should lead off your table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559673#action_12559673 ] Bryan Duxbury commented on HADOOP-2604: --- Sometimes, it'd be nice to iterate on the keys of a MapFile without actually reading the data. For instance, check out HStore#getClosestRowBefore - this seeks around checking keys and doesn't even use the value once it's been found. Every read to the underlying SequenceFile for that value is wasted. [hbase] Create an HBase-specific MapFile implementation --- Key: HADOOP-2604 URL: https://issues.apache.org/jira/browse/HADOOP-2604 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HBase uses the Hadoop MapFile class to store data persistently to disk. This is convenient, as it's already done (and maintained by other people :). However, it's beginning to look like there might be possible performance benefits to be had from doing an HBase-specific implementation of MapFile that incorporated some precise features. This issue should serve as a place to track discussion about what features might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2612) Mysterious ArrayOutOfBoundsException in HTable.commit
[
https://issues.apache.org/jira/browse/HADOOP-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559700#action_12559700
]
Bryan Duxbury commented on HADOOP-2612:
---
Can you reproduce this? Does it crash anything other than your client? If not,
this should probably be downgraded to a lower priority.
Mysterious ArrayOutOfBoundsException in HTable.commit
-
Key: HADOOP-2612
URL: https://issues.apache.org/jira/browse/HADOOP-2612
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Reporter: Michael Bieniosek
I got this exception using a post-0.15.0 hbase trunk:
Caused by: java.io.IOException: java.io.IOException:
java.lang.ArrayIndexOutOfBoundsException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:904)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:875)
at xxx.PutHbase$HbaseUploader.writeHbaseNoRetry(PutHbase.java:107)
Where writeHbaseNoRetry looks like:
private void writeHbaseNoRetry(HTable table, String column, String row,
File contents) throws IOException {
long lockid = table.startUpdate(new Text(row));
try {
table.put(lockid, new Text(column), FileUtil.readFile(contents));
table.commit(lockid);
} finally {
table.abort(lockid);
}
}
I found this in my error logs -- it is rare, and I am not sure how to
reproduce it. Contents could be 1kb-100kb long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559759#action_12559759 ] Bryan Duxbury commented on HADOOP-2443: --- I'm not sure I understand your logic. if I replaced line 231 with a continue, it would just continue the for loop that's iterating over the columns of the current result we pulled back. The reason that we break all the way out of the scanner iteration loop is because it is impossible for this scanner to ever yield another HRegionInfo that might match the current table. Continuing would only waste time traversing the rest of the scanner. This did, however, bring to my attention that if there are multiple .META. regions, and the table we're searching for is in the first .META. region, this code will actually search at least the first row of the second region too. So in reality, the break SCANNER_LOOP should be break REGION_LOOP. Does that make sense? [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559769#action_12559769 ] Bryan Duxbury commented on HADOOP-2443: --- You know what, I think you're right. I must have only tested it in scenarios when there are single tables. I've opened a new issue to track this problem, HADOOP-2631. I am resolving this issue again, because it's already been committed. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury resolved HADOOP-2443. --- Resolution: Fixed [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2611) [hbase] Create a RowMutation class in the public API
[hbase] Create a RowMutation class in the public API
Key: HADOOP-2611
URL: https://issues.apache.org/jira/browse/HADOOP-2611
Project: Hadoop
Issue Type: New Feature
Components: contrib/hbase
Reporter: Bryan Duxbury
Priority: Minor
Today, when you want to interact with a row in HBase, you start an update, make
changes, and then commit the lock. This is fine for very simple applications.
However, when you try to do things like support table operations as part of a
MapReduce job, it becomes more difficult to support.
I propose that we create a new class, RowMutation (a la the Bigtable paper),
which encapsulates a group of actions on a row, and make this available to API
consumers. It might look something like:
{code}
RowMutation r = table.getMutation(row_key);
r.setTimestamp();
r.put(new Text(colfam1:name, value));
r.delete(new Text(colfam2:deleted));
table.commit(r);
{code}
This syntax would supercede the existing startUpdate/commit format, which could
be deprecated and mapped to a RowMutation behind the scenes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2611) [hbase] Create a RowMutation class in the public API
[
https://issues.apache.org/jira/browse/HADOOP-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559143#action_12559143
]
Bryan Duxbury commented on HADOOP-2611:
---
Also note, this change would allow multiple concurrent updates to go on
against the same HTable instance at once, as the changes would be tracked in
the RowMutation instance, rather than internal to the HTable.
[hbase] Create a RowMutation class in the public API
Key: HADOOP-2611
URL: https://issues.apache.org/jira/browse/HADOOP-2611
Project: Hadoop
Issue Type: New Feature
Components: contrib/hbase
Reporter: Bryan Duxbury
Priority: Minor
Today, when you want to interact with a row in HBase, you start an update,
make changes, and then commit the lock. This is fine for very simple
applications. However, when you try to do things like support table
operations as part of a MapReduce job, it becomes more difficult to support.
I propose that we create a new class, RowMutation (a la the Bigtable paper),
which encapsulates a group of actions on a row, and make this available to
API consumers. It might look something like:
{code}
RowMutation r = table.getMutation(row_key);
r.setTimestamp();
r.put(new Text(colfam1:name, value));
r.delete(new Text(colfam2:deleted));
table.commit(r);
{code}
This syntax would supercede the existing startUpdate/commit format, which
could be deprecated and mapped to a RowMutation behind the scenes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2592) [hbase] Scanning, a region can let out a row that its not supposed to have
[ https://issues.apache.org/jira/browse/HADOOP-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2592: - Assignee: stack [hbase] Scanning, a region can let out a row that its not supposed to have -- Key: HADOOP-2592 URL: https://issues.apache.org/jira/browse/HADOOP-2592 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: stack Assignee: stack Priority: Critical Fix For: 0.16.0 Attachments: test3.patch, tti.patch Bryan's lazy cache patch was provoking a strange issue in TestTableIndex when flush size was small -- 64k instead of 1MB -- where a scan was returning random rows twice, w/ the second incidence coming out of a region whose range shouldn't even contain the row. After study, HalfMapFiles serving the top half of a parent region down in the daughter, can have rows in excess of the region endkey; scanning, the call to next will just return these rows w/ consideration of whether beyond region endkey. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2592) [hbase] Scanning, a region can let out a row that its not supposed to have
[ https://issues.apache.org/jira/browse/HADOOP-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2592: -- Fix Version/s: 0.16.0 Priority: Critical (was: Minor) This is a pretty critical issue, because it can result in getting an inconsistent answer from a scan. [hbase] Scanning, a region can let out a row that its not supposed to have -- Key: HADOOP-2592 URL: https://issues.apache.org/jira/browse/HADOOP-2592 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: stack Priority: Critical Fix For: 0.16.0 Attachments: test3.patch, tti.patch Bryan's lazy cache patch was provoking a strange issue in TestTableIndex when flush size was small -- 64k instead of 1MB -- where a scan was returning random rows twice, w/ the second incidence coming out of a region whose range shouldn't even contain the row. After study, HalfMapFiles serving the top half of a parent region down in the daughter, can have rows in excess of the region endkey; scanning, the call to next will just return these rows w/ consideration of whether beyond region endkey. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Attachment: 2512-v2.patch This patch passes all the tests locally. [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2512-v2.patch, 2513.patch Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Status: Patch Available (was: In Progress) Trying hudson. [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2512-v2.patch, 2513.patch Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559203#action_12559203 ] Bryan Duxbury commented on HADOOP-2604: --- I think it would make sense for us to maintain both a bloom filter and an index on row keys. That way, you can check the filter first to decide if you should check the index. Even if there's been deletions in the region that damage the filter, the index will still answer your question pretty quickly. We can maintain (re-create) the filter during compactions. I think we would see huge gains from having an always-on bloom filter, especially for sparser row spaces. [hbase] Create an HBase-specific MapFile implementation --- Key: HADOOP-2604 URL: https://issues.apache.org/jira/browse/HADOOP-2604 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HBase uses the Hadoop MapFile class to store data persistently to disk. This is convenient, as it's already done (and maintained by other people :). However, it's beginning to look like there might be possible performance benefits to be had from doing an HBase-specific implementation of MapFile that incorporated some precise features. This issue should serve as a place to track discussion about what features might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Open (was: Patch Available) Despite the fact that the tests pass, stack is right about the busted logic. I will fix that. I will also take the opportunity to convert that method to use the new MapFile.Reader getClosest (before) method to make the lookup more efficient. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2493.patch, regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Patch Available (was: Open) Trying Hudson. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2493-v2.patch, 2493-v3.patch, 2493.patch, regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Open (was: Patch Available) Forgot to include my new TestSplit2 in the patch hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2493-v2.patch, 2493-v3.patch, 2493.patch, regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Attachment: 2493-v3.patch Here's the forgotten test. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2493-v2.patch, 2493-v3.patch, 2493.patch, regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12559322#action_12559322 ] Bryan Duxbury commented on HADOOP-2604: --- It would also be nice to get a row count quickly from a MapFile, or at least check if the MapFile was empty. Can't do that today. [hbase] Create an HBase-specific MapFile implementation --- Key: HADOOP-2604 URL: https://issues.apache.org/jira/browse/HADOOP-2604 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HBase uses the Hadoop MapFile class to store data persistently to disk. This is convenient, as it's already done (and maintained by other people :). However, it's beginning to look like there might be possible performance benefits to be had from doing an HBase-specific implementation of MapFile that incorporated some precise features. This issue should serve as a place to track discussion about what features might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2600) [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before)
[ https://issues.apache.org/jira/browse/HADOOP-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2600: -- Attachment: 2600.patch This patch updates the code to do binary search instead of linear search. It makes my test suite pass in 18 minutes instead of 22. Sweet! [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before) - Key: HADOOP-2600 URL: https://issues.apache.org/jira/browse/HADOOP-2600 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2600.patch Once HADOOP-2532 is committed, we should make use of the new MapFile method to improve the performance of getClosestRow. This is an important improvement, because this method is used to look up region locations, a very critical part of HBase. Currently, the needed behavior is simulated by doing a linear scan of the MapFile. I think that 2532 will allow us to replace this with an index based lookup, which should make it much faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2600) [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before)
[ https://issues.apache.org/jira/browse/HADOOP-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2600: -- Status: Patch Available (was: Open) Trying Hudson [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before) - Key: HADOOP-2600 URL: https://issues.apache.org/jira/browse/HADOOP-2600 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2600.patch Once HADOOP-2532 is committed, we should make use of the new MapFile method to improve the performance of getClosestRow. This is an important improvement, because this method is used to look up region locations, a very critical part of HBase. Currently, the needed behavior is simulated by doing a linear scan of the MapFile. I think that 2532 will allow us to replace this with an index based lookup, which should make it much faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2597) [hbase] Performance - add a block cache
[ https://issues.apache.org/jira/browse/HADOOP-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12558634#action_12558634 ] Bryan Duxbury commented on HADOOP-2597: --- I like this idea. We could implement it with a wrapper around MapFile, something like CachingMapFile. [hbase] Performance - add a block cache --- Key: HADOOP-2597 URL: https://issues.apache.org/jira/browse/HADOOP-2597 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Tom White A block cache would cache fixed size blocks (default 64k) of data read from HDFS by the MapFile. It would help read performance for data close to recently read data (see Bigtable paper, section 6). It would be configurable on a per-column family basis. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[ https://issues.apache.org/jira/browse/HADOOP-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2598: -- Attachment: 2598.patch Actually, it looks like the message was in HConnectionManager$TableServers for checking the region location cache. This patch removes that as well as a commented logging line in HRegionServer. [hbase] Remove chatty debug logging from 2443 patch --- Key: HADOOP-2598 URL: https://issues.apache.org/jira/browse/HADOOP-2598 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Fix For: 0.16.0 Attachments: 2598.patch There was some leftover debug logging in HRegionServer in the patch for 2443. Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[ https://issues.apache.org/jira/browse/HADOOP-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2598: - Assignee: Bryan Duxbury [hbase] Remove chatty debug logging from 2443 patch --- Key: HADOOP-2598 URL: https://issues.apache.org/jira/browse/HADOOP-2598 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Fix For: 0.16.0 Attachments: 2598.patch There was some leftover debug logging in HRegionServer in the patch for 2443. Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[ https://issues.apache.org/jira/browse/HADOOP-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2598: -- Status: Patch Available (was: Open) Trying hudson. [hbase] Remove chatty debug logging from 2443 patch --- Key: HADOOP-2598 URL: https://issues.apache.org/jira/browse/HADOOP-2598 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Fix For: 0.16.0 Attachments: 2598.patch There was some leftover debug logging in HRegionServer in the patch for 2443. Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[hbase] Some minor improvements to changes in HADOOP-2443 - Key: HADOOP-2599 URL: https://issues.apache.org/jira/browse/HADOOP-2599 Project: Hadoop Issue Type: Improvement Reporter: Bryan Duxbury Priority: Minor Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); [ Show » ] stack - 13/Jan/08 10:54 PM Committed. Had some minor questions - see below - but it ain't enough to hold up patch and I'm afraid patch would rot if left hang (It touches lots of classes). Resolving. Thanks for the patch Bryan. Here's the couple of items: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury resolved HADOOP-2443. --- Resolution: Fixed Resolving issue. I have opened new issues for dealing with the problems mentioned in comments above. See HADOOP-2598 and HADOOP-2599. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2599: -- Component/s: contrib/hbase Assignee: Bryan Duxbury Description: Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); was: Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); [ Show » ] stack - 13/Jan/08 10:54 PM Committed. Had some minor questions - see below - but it ain't enough to hold up patch and I'm afraid patch would rot if left hang (It touches lots of classes). Resolving. Thanks for the patch Bryan. Here's the couple of items: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); [hbase] Some minor improvements to changes in HADOOP-2443 - Key: HADOOP-2599 URL: https://issues.apache.org/jira/browse/HADOOP-2599 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2600) [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before)
[hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before) - Key: HADOOP-2600 URL: https://issues.apache.org/jira/browse/HADOOP-2600 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Once HADOOP-2532 is committed, we should make use of the new MapFile method to improve the performance of getClosestRow. This is an important improvement, because this method is used to look up region locations, a very critical part of HBase. Currently, the needed behavior is simulated by doing a linear scan of the MapFile. I think that 2532 will allow us to replace this with an index based lookup, which should make it much faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12558708#action_12558708 ] Bryan Duxbury commented on HADOOP-2599: --- @stack: Regarding the third suggestion, are you indicating that we need a new class to hold a method that can append an unreasonably large timestamp onto a table name? Does the method seem complex enough to warrant that? If we create an HTableInfo class, what else would go in it? [hbase] Some minor improvements to changes in HADOOP-2443 - Key: HADOOP-2599 URL: https://issues.apache.org/jira/browse/HADOOP-2599 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[hbase] Remove chatty debug logging from 2443 patch --- Key: HADOOP-2598 URL: https://issues.apache.org/jira/browse/HADOOP-2598 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Fix For: 0.16.0 There was some leftover debug logging in HRegionServer in the patch for 2443. Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-2599 started by Bryan Duxbury. [hbase] Some minor improvements to changes in HADOOP-2443 - Key: HADOOP-2599 URL: https://issues.apache.org/jira/browse/HADOOP-2599 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2599.patch Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2599: -- Attachment: 2599.patch Here's a patch that implements the first two suggestions. [hbase] Some minor improvements to changes in HADOOP-2443 - Key: HADOOP-2599 URL: https://issues.apache.org/jira/browse/HADOOP-2599 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2599.patch Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ,,99); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2579) initializing a new HTable object against a nonexistent table throws a NoServerForRegionException instead of a TableNotFoundException when a different table has been creat
[
https://issues.apache.org/jira/browse/HADOOP-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Duxbury updated HADOOP-2579:
--
Attachment: 2579.patch
This patch includes Peter's new test cases and some fixes to HTable,
HConnectionManager, HRegionServer, HRegion, and HStore to correctly handle the
case when there's no tables at all in .META.
initializing a new HTable object against a nonexistent table throws a
NoServerForRegionException instead of a TableNotFoundException when a
different table has been created previously
---
Key: HADOOP-2579
URL: https://issues.apache.org/jira/browse/HADOOP-2579
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Reporter: Peter Dolan
Attachments: 2579.patch
When a table has been created, initializing a new HTable object for that
table works properly, as expected, but initializing a new HTable against a
nonexistent table in the same context (a table exists), a
NoServerForRegionException is thrown instead of TableNotFoundException, as
would be expected.
If there haven't been any tables created, initializing a new HTable object
against 'nosuchTable' throws TableNotFoundException, as expected.
Running the following TestCase exposes the issue on my machine.
{code:title=TestHTable.java|borderStyle=solid}
package org.apache.hadoop.hbase;
import java.io.IOException;
import org.apache.hadoop.io.Text;
/**
* Tests HTable
*/
public class TestHTable extends HBaseClusterTestCase implements HConstants {
public void testTableNotFoundExceptionWithoutAnyTables() {
try {
new HTable(conf, new Text(notATable));
fail(Should have thrown a TableNotFoundException);
} catch (TableNotFoundException e) {
// expected
} catch (IOException e) {
e.printStackTrace();
fail(Should have thrown a TableNotFoundException instead of a +
e.getClass());
}
}
public void testTableNotFoundExceptionWithATable() {
try {
HColumnDescriptor column =
new HColumnDescriptor(COLUMN_FAMILY.toString());
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor testTableADesc =
new HTableDescriptor(table);
testTableADesc.addFamily(column);
admin.createTable(testTableADesc);
// This should throw a TableNotFoundException, it has not been created
new HTable(conf, new Text(notATable));
fail(Should have thrown a TableNotFoundException);
} catch (TableNotFoundException e) {
// expected
} catch (IOException e) {
e.printStackTrace();
fail(Should have thrown a TableNotFoundException instead of a +
e.getClass());
}
}
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2579) initializing a new HTable object against a nonexistent table throws a NoServerForRegionException instead of a TableNotFoundException when a different table has been creat
[
https://issues.apache.org/jira/browse/HADOOP-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Duxbury updated HADOOP-2579:
--
Status: Patch Available (was: Open)
Tests pass locally, submitting to Hudson.
initializing a new HTable object against a nonexistent table throws a
NoServerForRegionException instead of a TableNotFoundException when a
different table has been created previously
---
Key: HADOOP-2579
URL: https://issues.apache.org/jira/browse/HADOOP-2579
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Reporter: Peter Dolan
Attachments: 2579.patch
When a table has been created, initializing a new HTable object for that
table works properly, as expected, but initializing a new HTable against a
nonexistent table in the same context (a table exists), a
NoServerForRegionException is thrown instead of TableNotFoundException, as
would be expected.
If there haven't been any tables created, initializing a new HTable object
against 'nosuchTable' throws TableNotFoundException, as expected.
Running the following TestCase exposes the issue on my machine.
{code:title=TestHTable.java|borderStyle=solid}
package org.apache.hadoop.hbase;
import java.io.IOException;
import org.apache.hadoop.io.Text;
/**
* Tests HTable
*/
public class TestHTable extends HBaseClusterTestCase implements HConstants {
public void testTableNotFoundExceptionWithoutAnyTables() {
try {
new HTable(conf, new Text(notATable));
fail(Should have thrown a TableNotFoundException);
} catch (TableNotFoundException e) {
// expected
} catch (IOException e) {
e.printStackTrace();
fail(Should have thrown a TableNotFoundException instead of a +
e.getClass());
}
}
public void testTableNotFoundExceptionWithATable() {
try {
HColumnDescriptor column =
new HColumnDescriptor(COLUMN_FAMILY.toString());
HBaseAdmin admin = new HBaseAdmin(conf);
HTableDescriptor testTableADesc =
new HTableDescriptor(table);
testTableADesc.addFamily(column);
admin.createTable(testTableADesc);
// This should throw a TableNotFoundException, it has not been created
new HTable(conf, new Text(notATable));
fail(Should have thrown a TableNotFoundException);
} catch (TableNotFoundException e) {
// expected
} catch (IOException e) {
e.printStackTrace();
fail(Should have thrown a TableNotFoundException instead of a +
e.getClass());
}
}
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[hbase] Create an HBase-specific MapFile implementation --- Key: HADOOP-2604 URL: https://issues.apache.org/jira/browse/HADOOP-2604 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HBase uses the Hadoop MapFile class to store data persistently to disk. This is convenient, as it's already done (and maintained by other people :). However, it's beginning to look like there might be possible performance benefits to be had from doing an HBase-specific implementation of MapFile that incorporated some precise features. This issue should serve as a place to track discussion about what features might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12558846#action_12558846 ] Bryan Duxbury commented on HADOOP-2604: --- Here's some of the ideas we're tossing around as a starter: * Exclude column family name from the file: Currently we store HStoreKeys, which are serialized to contain row, qualified cell name, and timestamp. However, seeing as how a given MapFile only ever belongs to one column family it's very wasteful to store the same column family name over and over again. In a custom implementation, we wouldn't have to save that data. * Separate indices for rows from qualified name and timestamp: Currently, the index in MapFiles is over all records, so the same row can appear in the index more than one time (differentiated by column name/timestamp). If the index just contained row keys, then we could store each row key exactly once, which would point to a record group of qualified names and timestamps (and values of course). Within the record group, there could be another separate small index on qualified name. This would again reduce the size of data stored, size of indices, and make it easier to do things like split regions lexically instead of skewed by cell count. * Use random rather than streaming reads: There is some indication that the existing MapFile implementation is optimized for streaming access; HBase supports random reads, which are therefore not efficient under MapFile. It would make sense for us to design our new implementation in such a way that it would be very cheap to do random access. [hbase] Create an HBase-specific MapFile implementation --- Key: HADOOP-2604 URL: https://issues.apache.org/jira/browse/HADOOP-2604 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HBase uses the Hadoop MapFile class to store data persistently to disk. This is convenient, as it's already done (and maintained by other people :). However, it's beginning to look like there might be possible performance benefits to be had from doing an HBase-specific implementation of MapFile that incorporated some precise features. This issue should serve as a place to track discussion about what features might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2493: - Assignee: Bryan Duxbury hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-2493 started by Bryan Duxbury. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Patch Available (was: In Progress) Passes locally, sending to Hudson. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2493.patch, regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Attachment: 2493.patch This patch adds a new test, TestSplit2, that makes sure a region of all the same row key will not be split. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2493.patch, regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2513: - Assignee: Bryan Duxbury [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Fix For: 0.16.0 Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-2513 started by Bryan Duxbury. [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Fix For: 0.16.0 Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Attachment: 2513.patch This patch includes some new tests in TestGet2 to highlight this problem and fixes in HStore to take care of the issues. [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2513.patch Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v9.patch Merged in Jim's patch in this version. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) Trying Hudson. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2591) [HQL] Addition of start command for execute the *.hql query file
[
https://issues.apache.org/jira/browse/HADOOP-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12558446#action_12558446
]
Bryan Duxbury commented on HADOOP-2591:
---
Feature freeze seems to be this Friday. Is this issue going to be completed by
then? If not, please adjust the fix version.
[HQL] Addition of start command for execute the *.hql query file
Key: HADOOP-2591
URL: https://issues.apache.org/jira/browse/HADOOP-2591
Project: Hadoop
Issue Type: New Feature
Components: contrib/hbase
Affects Versions: 0.16.0
Reporter: Edward Yoon
Assignee: Edward Yoon
Fix For: 0.16.0
{code}
$ cat queries.hql
CREATE TABLE webtable (
contents in_memory max_versions=10 compression=block,
anchor max_length=256 bloomfilter=counting_bloomfilter
vector_size=100 num_hash=4);
);
INSERT INTO webtable (contents, anchor:Hadoop)
VALUES ('html content', 'http://hadoop.apache.org')
WHERE row='http://www.hadoop.co.kr';
...
hql start ./queries.hql;
...
hql desc webtable;
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2542) Opening then closing a scanner delays the effects of a delete
[ https://issues.apache.org/jira/browse/HADOOP-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2542: -- Status: Open (was: Patch Available) Patch just contains a unit test that is intended to fail. Only need to submit a patch once it's ready to be tested for committing. Opening then closing a scanner delays the effects of a delete - Key: HADOOP-2542 URL: https://issues.apache.org/jira/browse/HADOOP-2542 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Clint Morgan Attachments: HADOOP-2542.patch I stumbled across the following odd behavior: When I open, then close a scanner on a table, then delete a row in the table, the delete does not work (i.e., I can still get the row). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) Tests failed by TestGet2 timing out. No idea what happened there, so I'll resubmit. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2584) Web UI displays an IOException instead of the Tables
[ https://issues.apache.org/jira/browse/HADOOP-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2584: -- Priority: Trivial (was: Major) This issue appears to be a cosmetic ui interface from what I can tell. I'm reducing the priority to match this problem. Web UI displays an IOException instead of the Tables Key: HADOOP-2584 URL: https://issues.apache.org/jira/browse/HADOOP-2584 Project: Hadoop Issue Type: Bug Components: contrib/hbase Affects Versions: 0.15.2 Reporter: Lars George Priority: Trivial For me after every second restart I get an error when loading the Hbase UI. Here the page: Master: 192.168.105.11:6 HQL, Local logs, Thread Dump, Log Level __ Master Attributes Attribute Name Value Description Filesystem lv1-xen-pdc-2.worldlingo.com:9000 Filesystem hbase is running on Hbase Root Directory /hbaseLocation of hbase home directory Online META Regions Name Server -ROOT-192.168.105.31:60020 .META.,,1 192.168.105.39:60020 Tables error msg : java.io.IOException: java.io.IOException: HStoreScanner failed construction at org.apache.hadoop.hbase.HStore$StoreFileScanner.(HStore.java:1879) at org.apache.hadoop.hbase.HStore$HStoreScanner.(HStore.java:2000) at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:1822) at org.apache.hadoop.hbase.HRegion$HScanner.(HRegion.java:1543) at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1118) at org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1465) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File does not exist: /hbase/hregion_1028785192/info/mapfiles/6628785818889695133/data at org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489) at org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSNamesystem.java:1380) at org.apache.hadoop.dfs.NameNode.getFileInfo(NameNode.java:425) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) Try the latest again. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v10.patch Ok, here's another go. The TestGet2 failure I had previously was actually correct, as I wasn't shutting down the test correctly. I fixed that, and in the process, merged in the changes from other recent patches. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v7.patch Another version, more cleaned up. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v8.patch Fixed javadoc warning. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: In Progress) Trying on hudson. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) Failure was in TestTableIndex in a place I'd never seen problems before. It was down in Flusher. I'm going to resubmit it and see if it passes again. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2500) [HBase] Unreadable region kills region servers
[ https://issues.apache.org/jira/browse/HADOOP-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12558036#action_12558036 ] Bryan Duxbury commented on HADOOP-2500: --- So, we should: * Change the no good message to something a tad more descriptive, like assignment of region is invalid * Enumerate the known ways that a RegionServer can fail to serve a region, trap those problems, and figure out what responses we'd like to give to those events [HBase] Unreadable region kills region servers -- Key: HADOOP-2500 URL: https://issues.apache.org/jira/browse/HADOOP-2500 Project: Hadoop Issue Type: Bug Components: contrib/hbase Environment: CentOS 5 Reporter: Chris Kline Priority: Critical Backgound: The name node (also a DataNode and RegionServer) in our cluster ran out of disk space. I created some space, restarted HDFS and fsck reported corruption with an HBase file. I cleared up that corruption and restarted HBase. I was still unable to read anything from HBase even though HSFS was now healthy. The following was gather from the log files. When HMaster starts up, it finds a region that is no good (Key: 17_125736271): 2007-12-24 09:07:14,342 DEBUG org.apache.hadoop.hbase.HMaster: Current assignment of spider_pages,17_125736271,1198286140018 is no good HMaster then assigns this region to RegionServer X.60: 2007-12-24 09:07:17,126 INFO org.apache.hadoop.hbase.HMaster: assigning region spider_pages,17_125736271,1198286140018 to server 10.100.11.60:60020 2007-12-24 09:07:20,152 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN : spider_pages,17_125736271,1198286140018 from 10.100.11.60:60020 The RegionServer has trouble reading that region (from the RegionServer log on X.60); Note that the worker thread exits 2007-12-24 09:07:22,611 DEBUG org.apache.hadoop.hbase.HStore: starting spider_pages,17_125736271,1198286140018/meta (2062710340/meta with reconstruction log: (/data/hbase1/hregion_2062710340/oldlogfile.log 2007-12-24 09:07:22,620 DEBUG org.apache.hadoop.hbase.HStore: maximum sequence id for hstore spider_pages,17_125736271,1198286140018/meta (2062710340/meta) is 4549496 2007-12-24 09:07:22,622 ERROR org.apache.hadoop.hbase.HRegionServer: error opening region spider_pages,17_125736271,1198286140018 java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readFully(DataInputStream.java:152) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1383) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1360) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1349) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1344) at org.apache.hadoop.hbase.HStore.doReconstructionLog(HStore.java:697) at org.apache.hadoop.hbase.HStore.init(HStore.java:632) at org.apache.hadoop.hbase.HRegion.init(HRegion.java:288) at org.apache.hadoop.hbase.HRegionServer.openRegion(HRegionServer.java:1211) at org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:1162) at java.lang.Thread.run(Thread.java:619) 2007-12-24 09:07:22,623 FATAL org.apache.hadoop.hbase.HRegionServer: Unhandled exception java.lang.NullPointerException at org.apache.hadoop.hbase.HRegionServer.reportClose(HRegionServer.java:1095) at org.apache.hadoop.hbase.HRegionServer.openRegion(HRegionServer.java:1217) at org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:1162) at java.lang.Thread.run(Thread.java:619) 2007-12-24 09:07:22,623 INFO org.apache.hadoop.hbase.HRegionServer: worker thread exiting The HMaster then tries to assign the same region to X.60 again and fails. The HMaster tries to assign the region to X.31 with the same result (X.31 worker thread exits). The file it is complaining about, /data/hbase1/hregion_2062710340/oldlogfile.log, is a zero-length file in HDFS. After deleting that file and restarting HBase, HBase appears to be back to normal. One thing I can't figure out is that the HMaster log show several entries after the worker thread on X.60 has exited suggesting that the RegionServer is talking with HMaster: 2007-12-24 09:08:23,349 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN : spider_pages,17_125736271,1198286140018 from 10.100.11.60:60020 2007-12-24 09:10:29,543 DEBUG org.apache.hadoop.hbase.HMaster: Received MSG_REPORT_PROCESS_OPEN : spider_pages,17_125736271,1198286140018 from 10.100.11.60:60020 There is no corresponding entry in the RegionServer's log. -- This message is
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v6.patch All the tests pass locally at this point, big improvement over the last version. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2555) Refactor the HTable#get and HTable#getRow methods to avoid repetition of retry-on-failure logic
[
https://issues.apache.org/jira/browse/HADOOP-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12558191#action_12558191
]
Bryan Duxbury commented on HADOOP-2555:
---
Generally, I love this patch. It'll nicely reduce the amount of copy-paste we
have.
I'd like to maybe take it one step further, though. Instead of the existing
callServerWithRetries, how about something like:
{code}
protected abstract class ServerCallableT implements CallableT {
HRegionLocation location;
HRegionInterface server;
Text row;
protected ServerCallable(Text row){
this.row = row;
}
void instantiateServer(boolean reload) throws IOException {
if (reload) {
tableServers = connection.reloadTableServers(tableName);
}
location = getRegionLocation(row);
server = connection.getHRegionConnection(location.getServerAddress());
}
}
protected T T getRegionServerWithRetries(ServerCallableT callable)
throws IOException, UnexpectedCallableException {
for(int tries = 0; tries numRetries; tries++) {
try {
callable.instantiateServer(tries == 0);
return callable.call();
} catch (IOException e) {
if (e instanceof RemoteException) {
e = RemoteExceptionHandler.decodeRemoteException((RemoteException) e);
}
if (tries == numRetries - 1) {
throw e;
}
if (LOG.isDebugEnabled()) {
LOG.debug(reloading table servers because: + e.getMessage());
}
} catch (Exception e) {
throw new UnexpectedCallableException(e);
}
try {
Thread.sleep(pause);
} catch (InterruptedException e) {
// continue
}
}
return null;
}
{code}
which takes us from
{code}
value = this.callServerWithRetries(new CallableMapWritable() {
public MapWritable call() throws IOException {
HRegionLocation r = getRegionLocation(row);
HRegionInterface server =
connection.getHRegionConnection(r.getServerAddress());
return server.getRow(r.getRegionInfo().getRegionName(), row, ts);
}
});
{code}
to
{code}
value = this.callServerWithRetries(new ServerCallableMapWritable(row) {
public MapWritable call() throws IOException {
return server.getRow(location.getRegionInfo().getRegionName(), row, ts);
}
});
{code}
This would save a few lines of code inside each internal block, move a little
more logic into the helper method, and generally jive better with the way my
HADOOP-2443 patch is going to need to work in the near future.
Comments?
Refactor the HTable#get and HTable#getRow methods to avoid repetition of
retry-on-failure logic
---
Key: HADOOP-2555
URL: https://issues.apache.org/jira/browse/HADOOP-2555
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Reporter: Peter Dolan
Priority: Minor
Attachments: hadoop-2555.patch
The following code is repeated in every one of HTable#get and HTable#getRow
methods:
{code:title=HTable.java|borderStyle=solid}
MapWritable value = null;
for (int tries = 0; tries numRetries; tries++) {
HRegionLocation r = getRegionLocation(row);
HRegionInterface server =
connection.getHRegionConnection(r.getServerAddress());
try {
value = server.getRow(r.getRegionInfo().getRegionName(), row, ts);
// This is the only line of code that changes significantly between methods
break;
} catch (IOException e) {
if (e instanceof RemoteException) {
e = RemoteExceptionHandler.decodeRemoteException((RemoteException)
e);
}
if (tries == numRetries - 1) {
// No more tries
throw e;
}
if (LOG.isDebugEnabled()) {
LOG.debug(reloading table servers because: + e.getMessage());
}
tableServers = connection.reloadTableServers(tableName);
}
try {
Thread.sleep(this.pause);
} catch (InterruptedException x) {
// continue
}
}
{code}
This should be factored out into a protected method that handles
retry-on-failure logic to facilitate more robust testing and the development
of new API methods.
Proposed modification:
// Execute the provided Callable against the server
protected T callServerWithRetries(CallableT callable) throws
RemoteException;
The above code could then be reduced to:
{code:title=HTable.java|borderStyle=solid}
MapWritable value = null;
final connection;
try {
value = callServerWithRetries(new CallableMapWritable() {
HRegionLocation r = getRegionLocation(row);
HRegionInterface server =
[jira] Updated: (HADOOP-2496) Snapshot of table
[ https://issues.apache.org/jira/browse/HADOOP-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2496: -- Fix Version/s: (was: 0.16.0) 0.17.0 Priority: Minor (was: Major) Adjusting the priority down to Minor, as this is new functionality. Also setting Fix Version to 0.17, as we have a lot of stuff to get done before 0.16 as it is. Snapshot of table - Key: HADOOP-2496 URL: https://issues.apache.org/jira/browse/HADOOP-2496 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Priority: Minor Fix For: 0.17.0 Havening an option to take a snapshot of a table would be vary useful in production. What I would like to see this option do is do a merge of all the data into one or more files stored in the same folder on the dfs. This way we could save data in case of a software bug in hadoop or user code. The other advantage would be to be able to export a table to multi locations. Say I had a read_only table that must be online. I could take a snapshot of it when needed and export it to a separate data center and have it loaded there and then i would have it online at multi data centers for load balancing and failover. I understand that hadoop takes the need out of havening backup to protect from failed servers, but this does not protect use from software bugs that might delete or alter data in ways we did not plan. We should have a way we can roll back a dataset. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v5.patch The latest version of the patch removes getTableServers and related methods entirely and switches HTable$ClientScanner to use getRegionLocation. Unfortunately, I now have TestTableIndex and TestTableMapReduce failing all the time. The error seem to be related to scanners acting strangely with rows in the wrong regions, but I can't be sure of what's going on yet. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v4.patch All tests are passing locally. This patch includes caching, so performance should be comparable to what you see with the other region location lookup approach. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch, 2443-v4.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2532) Add to MapFile a getClosest that returns key that comes just-before if key not present (Currently does just-after only).
[ https://issues.apache.org/jira/browse/HADOOP-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12557439#action_12557439 ] Bryan Duxbury commented on HADOOP-2532: --- +1 This should make my HBase cache patch much more efficient. Add to MapFile a getClosest that returns key that comes just-before if key not present (Currently does just-after only). Key: HADOOP-2532 URL: https://issues.apache.org/jira/browse/HADOOP-2532 Project: Hadoop Issue Type: New Feature Reporter: stack Assignee: stack Priority: Minor Fix For: 0.16.0 Attachments: getclosestbefore-v2.patch, getclosestbefore-v3.patch, getclosestbefore.patch The list of regions that make up a table in hbase are effectively kept in a mapfile. Regions are identified by the first row contained by that region. To find the region that contains a particular row, we need to be able to search the mapfile of regions to find the closest matching row that falls just-before the searched-for key rather than the just-after that is current mapfile getClosest behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2478) [hbase] restructure how HBase lays out files in the file system
[
https://issues.apache.org/jira/browse/HADOOP-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12557514#action_12557514
]
Bryan Duxbury commented on HADOOP-2478:
---
If this won't be fixed until 0.17, which is months away, should we apply my
HStore refactor patch in the meantime?
[hbase] restructure how HBase lays out files in the file system
---
Key: HADOOP-2478
URL: https://issues.apache.org/jira/browse/HADOOP-2478
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Affects Versions: 0.17.0
Reporter: Jim Kellerman
Assignee: Jim Kellerman
Fix For: 0.17.0
Attachments: patch.txt
Currently HBase has a pretty flat directory structure. For example:
{code}
/hbase/hregion_70236052/info
/hbase/hregion_70236052/info/info/4328260619704027575
/hbase/hregion_70236052/info/mapfiles/4328260619704027575
/hbase/hregion_70236052/info/mapfiles/4328260619704027575/data
/hbase/hregion_70236052/info/mapfiles/4328260619704027575/index
{code}
All the region directories are under the root directory, and with encoded
region names, it is impossible to determine what table a region belongs to.
This should be restructured to:
{code}
/root-directory/table-name/encoded-region-name/column-family/{info,mapfiles}
{code}
It will be necessary to provide a migration script from current trunk to the
new structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v3.patch Not quite done yet - a few tests don't pass, and there is no caching at all right now. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: 2443-v3.patch Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2364) when hbase regionserver restarts, it says impossible state for createLease()
[ https://issues.apache.org/jira/browse/HADOOP-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12556655#action_12556655 ] Bryan Duxbury commented on HADOOP-2364: --- Did this actually cause any problems? Did the Region Server continue to start up ok, or is this just a spurious error message? when hbase regionserver restarts, it says impossible state for createLease() -- Key: HADOOP-2364 URL: https://issues.apache.org/jira/browse/HADOOP-2364 Project: Hadoop Issue Type: Bug Components: contrib/hbase Affects Versions: 0.16.0 Reporter: Michael Bieniosek Priority: Minor I restarted a regionserver, and got this error in its logs: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.AssertionError: Impossible state for createLease(): Lease -435227488/-435227488 is still held. at org.apache.hadoop.hbase.Leases.createLease(Leases.java:145) at org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1278 ) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596) at org.apache.hadoop.ipc.Client.call(Client.java:482) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184) at $Proxy0.regionServerStartup(Unknown Source) at org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.jav a:1025) at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:659) at java.lang.Thread.run(Unknown Source) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2405) [hbase] Merge region tool exposed in shell and/or in UI
[ https://issues.apache.org/jira/browse/HADOOP-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12556694#action_12556694 ] Bryan Duxbury commented on HADOOP-2405: --- So, you envision the merge operation to not only require manual triggering but to require manual targeting? Shouldn't the point of merging regions be to maintain the equilibrium of size of regions? Under what circumstances will you have to manually intervene to keep regions appropriately sized? It seems like this should really only happen after a substantial number of deletions has occurred, right? That would cause a compacted region to shrink below a healthy size, and if it could be merged with a neighbor, it would be nice. This logic should be built in and automatic, otherwise it would require constant monitoring of region sizes by an administrator. Other than this sort of automatic merging, when would you want to manually merge two regions? Doesn't that expose a somewhat dangerous amount of functionality to the end user? [hbase] Merge region tool exposed in shell and/or in UI --- Key: HADOOP-2405 URL: https://issues.apache.org/jira/browse/HADOOP-2405 Project: Hadoop Issue Type: New Feature Components: contrib/hbase Reporter: stack Priority: Minor hbase has support for merging regions. Expose a merge trigger in the shell or in the UI (Can only merge adjacent features so perhaps only makes sense in UI in the regionserver UI). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Fix Version/s: 0.16.0 [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Fix For: 0.16.0 Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Fix Version/s: 0.16.0 [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Fix For: 0.16.0 Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Fix Version/s: 0.16.0 hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Priority: Critical Fix For: 0.16.0 Attachments: regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2545) hbase rest server should be started with hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HADOOP-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2545: -- Priority: Minor (was: Major) hbase rest server should be started with hbase-daemon.sh Key: HADOOP-2545 URL: https://issues.apache.org/jira/browse/HADOOP-2545 Project: Hadoop Issue Type: New Feature Components: contrib/hbase Reporter: Michael Bieniosek Priority: Minor Currently, the hbase rest server is started with the hbase script. But it should be started with the hbase-daemon script, which allows better configuration options. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2545) hbase rest server should be started with hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HADOOP-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12556805#action_12556805 ] Bryan Duxbury commented on HADOOP-2545: --- Can you suggest what exactly you'd like to see change in the way the REST servlet is started? What advantages do you gain? hbase rest server should be started with hbase-daemon.sh Key: HADOOP-2545 URL: https://issues.apache.org/jira/browse/HADOOP-2545 Project: Hadoop Issue Type: New Feature Components: contrib/hbase Reporter: Michael Bieniosek Priority: Minor Currently, the hbase rest server is started with the hbase script. But it should be started with the hbase-daemon script, which allows better configuration options. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2521) [hbase] HStoreFiles needlessly store the column family name in every entry
[ https://issues.apache.org/jira/browse/HADOOP-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12555997#action_12555997 ] Bryan Duxbury commented on HADOOP-2521: --- Jim, I am aware that multiple qualified cells show up in the same store file per row in the same HStoreFile. I'm just suggesting that the part that comes before the qualified name is unnecessary. I understand that changing this would necessitate adding a new key type and transform logic, but I'm not convinced that the translation would actually cost that much more time. You have to recognize that even though the data is precomputed, it is probably coming off of a disk on another computer through the network in 64MB blocks. I have to think that the added transmission time of all the redundant data in aggregate is at least as much as the added time it would take to do translation, and possibly more. [hbase] HStoreFiles needlessly store the column family name in every entry -- Key: HADOOP-2521 URL: https://issues.apache.org/jira/browse/HADOOP-2521 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
[ https://issues.apache.org/jira/browse/HADOOP-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12556020#action_12556020 ] Bryan Duxbury commented on HADOOP-2518: --- After some discussion with Jim, it looks like this patch is going to be put on hold. After he's checked in his patch to change file organization, we can re-address this desire. [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage -- Key: HADOOP-2518 URL: https://issues.apache.org/jira/browse/HADOOP-2518 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2518.patch HStore, the class responsible for managing data storage for column families within a region, is a very large class with more than a few internal classes. I propose moving HStore and HStoreFile into the hbase.store subpackage, and then moving internal classes out of HStore and into their own package-internal classes within the subpackage. This will make the code much more readable and manageable. In addition, it will help to logically delineate which classes are involved in managing a store. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
[hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage -- Key: HADOOP-2518 URL: https://issues.apache.org/jira/browse/HADOOP-2518 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor HStore, the class responsible for managing data storage for column families within a region, is a very large class with more than a few internal classes. I propose moving HStore and HStoreFile into the hbase.store subpackage, and then moving internal classes out of HStore and into their own package-internal classes within the subpackage. This will make the code much more readable and manageable. In addition, it will help to logically delineate which classes are involved in managing a store. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
[ https://issues.apache.org/jira/browse/HADOOP-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2518: -- Attachment: 2518.patch Here's my go at the refactor described above. Files modified: D src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHStoreFile.java D src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHMemcache.java A src/contrib/hbase/src/test/org/apache/hadoop/hbase/store A src/contrib/hbase/src/test/org/apache/hadoop/hbase/store/TestHStoreFile.java A src/contrib/hbase/src/test/org/apache/hadoop/hbase/store/TestHMemcache.java M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestCompaction.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogKey.java D src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java D src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/BloomFilterDescriptor.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLog.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/CompactionReader.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStoreSize.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/Memcache.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStore.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStoreFile.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/StoreFileScanner.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/MapFileCompactionReader.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStoreScanner.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HAbstractScanner.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage -- Key: HADOOP-2518 URL: https://issues.apache.org/jira/browse/HADOOP-2518 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2518.patch HStore, the class responsible for managing data storage for column families within a region, is a very large class with more than a few internal classes. I propose moving HStore and HStoreFile into the hbase.store subpackage, and then moving internal classes out of HStore and into their own package-internal classes within the subpackage. This will make the code much more readable and manageable. In addition, it will help to logically delineate which classes are involved in managing a store. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2521) [hbase] HStoreFiles needlessly store the column family name in every entry
[hbase] HStoreFiles needlessly store the column family name in every entry -- Key: HADOOP-2521 URL: https://issues.apache.org/jira/browse/HADOOP-2521 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2504) [hbase] REST servlet method for deleting a scanner was not properly mapped
[ https://issues.apache.org/jira/browse/HADOOP-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2504: -- Status: Open (was: Patch Available) [hbase] REST servlet method for deleting a scanner was not properly mapped -- Key: HADOOP-2504 URL: https://issues.apache.org/jira/browse/HADOOP-2504 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2504-2.patch, 2504.patch As reported by Billy Pearson, the code for DELETE on an existing scanner was written, but not properly mapped to real methods after the REST refactoring work was done. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2504) [hbase] REST servlet method for deleting a scanner was not properly mapped
[ https://issues.apache.org/jira/browse/HADOOP-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2504: -- Status: Patch Available (was: Open) Resending to Hudson, as it didn't seem to get built. [hbase] REST servlet method for deleting a scanner was not properly mapped -- Key: HADOOP-2504 URL: https://issues.apache.org/jira/browse/HADOOP-2504 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2504-2.patch, 2504.patch As reported by Billy Pearson, the code for DELETE on an existing scanner was written, but not properly mapped to real methods after the REST refactoring work was done. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the now timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at now time. Instead, you will get the value written into the past cell, because even though it is tagged as having happened in the past, it actually *was written* after the now cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2507) [hbase] REST servlet does not properly base64 row keys and column names
[ https://issues.apache.org/jira/browse/HADOOP-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2507: -- Status: Patch Available (was: Open) Sending to Hudson [hbase] REST servlet does not properly base64 row keys and column names --- Key: HADOOP-2507 URL: https://issues.apache.org/jira/browse/HADOOP-2507 Project: Hadoop Issue Type: Bug Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2507.patch At the moment, the REST servlet treats row keys and column names as though they are all printable, when in fact they can definitely be arbitrary binary strings. When returning these items, they should be base64 encoded like cell values are. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2347) [hbase] REST servlet not thread safe but run in a threaded manner
[ https://issues.apache.org/jira/browse/HADOOP-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12555212 ] Bryan Duxbury commented on HADOOP-2347: --- It's committed. It should be in any nightly since December 4th. [hbase] REST servlet not thread safe but run in a threaded manner - Key: HADOOP-2347 URL: https://issues.apache.org/jira/browse/HADOOP-2347 Project: Hadoop Issue Type: Bug Reporter: Bryan Duxbury Assignee: Bryan Duxbury Fix For: 0.16.0 Attachments: rest-threadsafe.patch Jetty serves the REST servlet in a threaded manner, but the servlet has some thread safety issues. These need to be resolved. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12555126 ] Bryan Duxbury commented on HADOOP-2493: --- If I understand correctly, this problem only really manifests when the size of an individual row is bigger than the size of a region. Does that make sense? If so, a solution would be to check the row count of a region when splitting, and if it's 1, don't split. That way, you wouldn't end up with two regions with the same start/end info. hbase will split on row when the start and end row is the same cuase data loss -- Key: HADOOP-2493 URL: https://issues.apache.org/jira/browse/HADOOP-2493 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Priority: Critical Attachments: regions_shot.JPG While testing hbase splits with my code I was loading a table to become a inverted index on some links I was using the anchor text as the row key and the column parent:child as url:(siteurl) and the data is the count of the links pointing to the siteurl with row key anchor text. but a lot of sites have image links and I use image as the anchor text for my testing code so there is a lot of image links. I changed the max file size of hbase to 16mb for testing and have been able to recreate the same error. When the table get big it splits on the column image as the end key for one table and the start of the next table later it splits to where the start key and end key was image for one of the splits. After that it keep spiting the region with start key as image and the end key the same. So I have multi splits with start key and end key as image unless the master keeps track of the row key and partend:child data on the splits I do not thank all the data will get returned when querying it. I have attached a screen shot of my regions i thank there should be some logic to where if the start and end row key is the same the region does not split or we need to start keeping track of the start key, column data on the master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12555151 ] Bryan Duxbury commented on HADOOP-2443: --- I think I have a pretty good strategy in mind, but I've sort of hit a snag. The problem is that when you're trying to determine which region a key lives in, the list of regions is keyed by the region's start key, and you're looking for the region that is closest to (without exceeding) the search key. That is, if you have regions [0, 10, 20] and you're trying to figure out where key 15 lives, you want the answer to be 10. Then data is all there for this sort of thing, but the interface of MapFile, which is where the .META. table is ultimately stored, doesn't have anything like this. What we need essentially is a method just like getClosest, but that returns the last key found after the one the goes over the search key. The alternative is doing a linear search of some sort manually, which would be pretty inefficient. [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list Key: HADOOP-2443 URL: https://issues.apache.org/jira/browse/HADOOP-2443 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Assignee: Bryan Duxbury Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table. Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load. Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time. Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2507) [hbase] REST servlet does not properly base64 row keys and column names
[hbase] REST servlet does not properly base64 row keys and column names --- Key: HADOOP-2507 URL: https://issues.apache.org/jira/browse/HADOOP-2507 Project: Hadoop Issue Type: Bug Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor At the moment, the REST servlet treats row keys and column names as though they are all printable, when in fact they can definitely be arbitrary binary strings. When returning these items, they should be base64 encoded like cell values are. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2507) [hbase] REST servlet does not properly base64 row keys and column names
[ https://issues.apache.org/jira/browse/HADOOP-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2507: -- Attachment: 2507.patch This patch adds base64 encoding to column names and row key values in both GET requests and scanner GET requests. [hbase] REST servlet does not properly base64 row keys and column names --- Key: HADOOP-2507 URL: https://issues.apache.org/jira/browse/HADOOP-2507 Project: Hadoop Issue Type: Bug Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2507.patch At the moment, the REST servlet treats row keys and column names as though they are all printable, when in fact they can definitely be arbitrary binary strings. When returning these items, they should be base64 encoded like cell values are. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2503) REST Insert / Select
[ https://issues.apache.org/jira/browse/HADOOP-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2503: - Assignee: Bryan Duxbury REST Insert / Select Key: HADOOP-2503 URL: https://issues.apache.org/jira/browse/HADOOP-2503 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Billy Pearson Assignee: Bryan Duxbury Priority: Critical Fix For: 0.16.0 Attachments: 2503.patch On inserts I submit the row key as urlencoded and its stored that way with the urlencodeing intact. But when I select with rest I send the row key as urlencoded as I stored it and get nothing back. I thank the problem is the rest interfase is urldecodeing the request before looking up the data in the table. I can select with the urlencoded row key with the shell and get results. This makes data not selectable if it has any urlencodeing chrs. So 1 of two options I see Make the insert store the row key after urldecode or make the select look up the row key the way it is submitted with out urldecodeing the key. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2504) [hbase] REST servlet method for deleting a scanner was not properly mapped
[ https://issues.apache.org/jira/browse/HADOOP-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2504: -- Status: Patch Available (was: Open) Sending to Hudson. [hbase] REST servlet method for deleting a scanner was not properly mapped -- Key: HADOOP-2504 URL: https://issues.apache.org/jira/browse/HADOOP-2504 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2504-2.patch, 2504.patch As reported by Billy Pearson, the code for DELETE on an existing scanner was written, but not properly mapped to real methods after the REST refactoring work was done. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2504) [hbase] REST servlet method for deleting a scanner was not properly mapped
[ https://issues.apache.org/jira/browse/HADOOP-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2504: -- Attachment: 2504.patch This should fix the problem. [hbase] REST servlet method for deleting a scanner was not properly mapped -- Key: HADOOP-2504 URL: https://issues.apache.org/jira/browse/HADOOP-2504 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Attachments: 2504.patch As reported by Billy Pearson, the code for DELETE on an existing scanner was written, but not properly mapped to real methods after the REST refactoring work was done. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
