[hbase] Refactoring idea: Move classes into Client, Master, and RegionServer packages
All, I've been crawling around the HBase codebase for a little while now, and I think I have a proposal that would make it easer to find your way around in the codebase in general. I think that we should make three new packages below org.apache.hadoop.hbase, client, master, and regionserver. The client package would contain HTable, client-side scanning stuff, HBaseAdmin, the MapReduce-related stuff, the shell, REST and Thrift. The master package would contain HMaster, maybe Leases, any other classes that belong to master. The regionserver package would contain HRegionServer, HRegion, HStore and all its subclasses (HStoreFile, etc). Whatever is left over should be stuff that's pretty common to all the sub-packages, so we can either leave that in the hbase package, or push it down into a common subpackage. This would make it much easier for new contributors to decide where to look for stuff, as well as make it more obvious what the architectural divisions of the system are. To boot, it would allow us to reorganize our tests into similar subpackages, which has the advantage of allowing us to think about, for instance, client tests passing/failing as a group, rather than scattered alphabetically throughout the entire suite. This idea would probably erase HADOOP-2518, or at least change the goal to factor HStore down into o.a.h.h.regionserver.store. Comments? -Bryan
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Affects Version/s: 0.16.0 Fix Version/s: (was: 0.16.0) 0.17.0 Putting this off until later, as the performance hit is pretty substantial, and I think there might be a solution that gives us both performance and correctness. > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Affects Versions: 0.16.0 > Reporter: Bryan Duxbury >Assignee: stack > Fix For: 0.17.0 > > Attachments: 2512-v2.patch, 2513.patch > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560734#action_12560734 ] Bryan Duxbury commented on HADOOP-2513: --- I understand that the performance degrades as a result of this patch, but I'm much more worried about the correctness of the data than the performance. Sure, it might make it faster to leave this out, but the first time someone happens to hit this problem, they'll go out of their minds trying to track it down. I think we can make some improvements that will make this patch a little faster. I will take another look and try to freshen up the patch. > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: stack > Fix For: 0.16.0 > > Attachments: 2512-v2.patch, 2513.patch > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2666) [hbase] Cleanup: All generated code should go into the o.a.h.h.generated package
[ https://issues.apache.org/jira/browse/HADOOP-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560733#action_12560733 ] Bryan Duxbury commented on HADOOP-2666: --- -1 on this one. It seems like it would be a little confusing. If there's a thrift package, then the generated code for thrift should probably be under it. > [hbase] Cleanup: All generated code should go into the o.a.h.h.generated > package > > > Key: HADOOP-2666 > URL: https://issues.apache.org/jira/browse/HADOOP-2666 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: stack >Priority: Trivial > > As is, generated code goes into o.a.h.h.thrift.generated, > o.a.h.h.hql.generated, etc. Rather, have it all go into the one package, > o.a.h.h.generated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2667) [hbase] Clients should be able to open an OutputStream for putting a value into a table
[ https://issues.apache.org/jira/browse/HADOOP-2667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560731#action_12560731 ] Bryan Duxbury commented on HADOOP-2667: --- I'm not sure that streaming multi-gigabyte blobs into HBase really makes any sense. If you're going to have a lot of multi-gigabyte blobs, HDFS is probably a better fit for you, and just store the metadata in HBase. However, I guess I could see situations where you might want to stream something like a few megabytes rather than keep it all in memory. However, as I understand it, the current RPC system probably wouldn't support this very well, so it would bear some scrutiny. > [hbase] Clients should be able to open an OutputStream for putting a value > into a table > --- > > Key: HADOOP-2667 > URL: https://issues.apache.org/jira/browse/HADOOP-2667 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: stack >Priority: Minor > > > Right now the only interface for insertion is via a byte[], which severely > > limits the flexibility of the system. Added 2007-10-08 by stuhood. > Tell us more Stu why you need this feature? (Because you want to write BLOBs > of multiple Gigabytes into an hbase cell?) > Added as part of migrating new feature requests from the obsoleted > http://wiki.apache.org/hadoop/Hbase/HbaseFeatureRequests -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Status: Patch Available (was: Open) Trying Hudson. > [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or > table you are enumerating isn't the first table > > > Key: HADOOP-2631 > URL: https://issues.apache.org/jira/browse/HADOOP-2631 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: Bryan Duxbury > Attachments: 2631-v2.patch, 2631.patch > > > Clint Morgan noticed that the current HTable#getStartKeys implementation > isn't correct when there are more than one table, or the table you're trying > to get the start keys of isn't the only table. I believe that the solution is > making sure that the scanner starts at the key that should lead off your > table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Attachment: 2631-v2.patch New TestGetStartKeys was unreliable in last patch. This one passes consistently. > [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or > table you are enumerating isn't the first table > > > Key: HADOOP-2631 > URL: https://issues.apache.org/jira/browse/HADOOP-2631 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: Bryan Duxbury > Attachments: 2631-v2.patch, 2631.patch > > > Clint Morgan noticed that the current HTable#getStartKeys implementation > isn't correct when there are more than one table, or the table you're trying > to get the start keys of isn't the only table. I believe that the solution is > making sure that the scanner starts at the key that should lead off your > table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Attachment: 2631.patch This patch seems to resolve the problem. It still breaks out of the scanner loop when the table name doesn't match the table we're searching for, but it also starts the scanner at the name of the table we're searching for instead of EMPTY_START_ROW, which was causing rows for different tables to pop up first and prematurely end the scan. > [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or > table you are enumerating isn't the first table > > > Key: HADOOP-2631 > URL: https://issues.apache.org/jira/browse/HADOOP-2631 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury >Assignee: Bryan Duxbury > Attachments: 2631.patch > > > Clint Morgan noticed that the current HTable#getStartKeys implementation > isn't correct when there are more than one table, or the table you're trying > to get the start keys of isn't the only table. I believe that the solution is > making sure that the scanner starts at the key that should lead off your > table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[ https://issues.apache.org/jira/browse/HADOOP-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2631: -- Component/s: contrib/hbase > [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or > table you are enumerating isn't the first table > > > Key: HADOOP-2631 > URL: https://issues.apache.org/jira/browse/HADOOP-2631 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: Bryan Duxbury > > Clint Morgan noticed that the current HTable#getStartKeys implementation > isn't correct when there are more than one table, or the table you're trying > to get the start keys of isn't the only table. I believe that the solution is > making sure that the scanner starts at the key that should lead off your > table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2637) [hbase] WrongRegionException looks suspiciously like client region selection is doing its calc. wrong
[
https://issues.apache.org/jira/browse/HADOOP-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559975#action_12559975
]
Bryan Duxbury commented on HADOOP-2637:
---
Just so I know I'm reading the exception correctly, this indicates that it's
trying to operate on a region that spans from 103481 to 163328, and the
row we're trying to commit on is actually the end key, right?
Well, looking over the location caching code, it can't be that, because it
specifically tests for < endKey. That means that you had to get this wrong
region directly from the .META. region. The methods for doing that don't check
to see if the key you are trying to locate is > endKey of the regioninfo we
bring back. We could add this check, but I'm not sure it would be much help.
We're using getClosestRowBefore to figure out what key in the .META. to query
for region info, and if it's coming back with the region immediately before it,
the only thing I could think that would be happening is it's in the middle of a
write or something and hasn't written the new regioninfo row yet. Otherwise, it
should come up.
Did you see lots of these errors, or just one? Not that it's excusable at any
rate, but if it's pretty frequent, it'd mean there was a much larger problem
than if this was incidental.
> [hbase] WrongRegionException looks suspiciously like client region selection
> is doing its calc. wrong
> -
>
> Key: HADOOP-2637
> URL: https://issues.apache.org/jira/browse/HADOOP-2637
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
>Reporter: stack
>Priority: Minor
>
> I got this this evening:
> {code}
> 08/01/17 05:57:06 ERROR hbase.PerformanceEvaluation: Failed
> org.apache.hadoop.hbase.WrongRegionException:
> org.apache.hadoop.hbase.WrongRegionException: Requested row out of range for
> HRegion TestTable,103481,1200548919843, startKey='103481',
> getEndKey()='163328', row='163328'
> at org.apache.hadoop.hbase.HRegion.checkRow(HRegion.java:1491)
> at org.apache.hadoop.hbase.HRegion.obtainRowLock(HRegion.java:1536)
> at org.apache.hadoop.hbase.HRegion.batchUpdate(HRegion.java:1231)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1497)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:908)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:968)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:939)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:461)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:329)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:525)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runNIsOne(PerformanceEvaluation.java:546)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:568)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:663)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:682)
> ..
> {code}
> We seem to be asking for the endkey on a region rather than asking for the
> next region, the one that has the asked-for row as its start key.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury resolved HADOOP-2443. --- Resolution: Fixed > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, > 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559769#action_12559769 ] Bryan Duxbury commented on HADOOP-2443: --- You know what, I think you're right. I must have only tested it in scenarios when there are single tables. I've opened a new issue to track this problem, HADOOP-2631. I am resolving this issue again, because it's already been committed. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, > 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2631) [hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table
[hbase] 2443 breaks HTable.getStartKeys when there is more than one table or table you are enumerating isn't the first table Key: HADOOP-2631 URL: https://issues.apache.org/jira/browse/HADOOP-2631 Project: Hadoop Issue Type: Bug Reporter: Bryan Duxbury Assignee: Bryan Duxbury Clint Morgan noticed that the current HTable#getStartKeys implementation isn't correct when there are more than one table, or the table you're trying to get the start keys of isn't the only table. I believe that the solution is making sure that the scanner starts at the key that should lead off your table. We should create a TestGetStartKeys test to verify this behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559759#action_12559759 ] Bryan Duxbury commented on HADOOP-2443: --- I'm not sure I understand your logic. if I replaced line 231 with a continue, it would just continue the for loop that's iterating over the columns of the current result we pulled back. The reason that we break all the way out of the scanner iteration loop is because it is impossible for this scanner to ever yield another HRegionInfo that might match the current table. Continuing would only waste time traversing the rest of the scanner. This did, however, bring to my attention that if there are multiple .META. regions, and the table we're searching for is in the first .META. region, this code will actually search at least the first row of the second region too. So in reality, the break SCANNER_LOOP should be break REGION_LOOP. Does that make sense? > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, > 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2612) Mysterious ArrayOutOfBoundsException in HTable.commit
[
https://issues.apache.org/jira/browse/HADOOP-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559700#action_12559700
]
Bryan Duxbury commented on HADOOP-2612:
---
Can you reproduce this? Does it crash anything other than your client? If not,
this should probably be downgraded to a lower priority.
> Mysterious ArrayOutOfBoundsException in HTable.commit
> -
>
> Key: HADOOP-2612
> URL: https://issues.apache.org/jira/browse/HADOOP-2612
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
>Reporter: Michael Bieniosek
>
> I got this exception using a post-0.15.0 hbase trunk:
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
> Source)
> at java.lang.reflect.Constructor.newInstance(Unknown Source)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:904)
> at org.apache.hadoop.hbase.HTable.commit(HTable.java:875)
> at xxx.PutHbase$HbaseUploader.writeHbaseNoRetry(PutHbase.java:107)
> Where writeHbaseNoRetry looks like:
> private void writeHbaseNoRetry(HTable table, String column, String row,
> File contents) throws IOException {
> long lockid = table.startUpdate(new Text(row));
> try {
> table.put(lockid, new Text(column), FileUtil.readFile(contents));
> table.commit(lockid);
> } finally {
> table.abort(lockid);
> }
> }
> I found this in my error logs -- it is rare, and I am not sure how to
> reproduce it. Contents could be 1kb-100kb long.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559673#action_12559673 ] Bryan Duxbury commented on HADOOP-2604: --- Sometimes, it'd be nice to iterate on the keys of a MapFile without actually reading the data. For instance, check out HStore#getClosestRowBefore - this seeks around checking keys and doesn't even use the value once it's been found. Every read to the underlying SequenceFile for that value is wasted. > [hbase] Create an HBase-specific MapFile implementation > --- > > Key: HADOOP-2604 > URL: https://issues.apache.org/jira/browse/HADOOP-2604 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Priority: Minor > > Today, HBase uses the Hadoop MapFile class to store data persistently to > disk. This is convenient, as it's already done (and maintained by other > people :). However, it's beginning to look like there might be possible > performance benefits to be had from doing an HBase-specific implementation of > MapFile that incorporated some precise features. > This issue should serve as a place to track discussion about what features > might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Subclassing SequenceFile and MapFile
We have HADOOP-2604 open, which is about creating a brand-new MapFile implementation with HBase's needs specifically in mind. Some of the things we'd like to do there (different indexing schemes, etc) seem like they would be hard to implement with the existing MapFile even if we had the ability to subclass it better. I think in terms of easing ongoing HBase development, it would make sense to go in our own direction. -Bryan On Jan 16, 2008, at 11:14 AM, Jim Kellerman wrote: HBase has several subclasses of MapFile already: org.apache.hadoop.hbase.HStoreFile$ HbaseMapFile BloomFilterMapFile HalfMapFileReader If MapFile were more subclassable (had protected members instead of private or accessor methods) we would probably add client side caching, bloom filters (to determine if a key exists in a map file - different from BloomFilterMapFile above which is a mix-in of MapFile and BloomFilter) Tom White said (in https://issues.apache.org/jira/browse/HADOOP-2604) If MapFile.Reader were an interface (or an abstract class with a no args constructor) then BloomFilterMapFile.Reader, HalfMapFileReader and caching Readers could be implemented as wrappers instead of in a static hierarchy. This would make it easier to mix and match readers (e.g. with or without caching) without passing all possible parameters in the constructor. So we'd like to make MapFile (and probably SequenceFile) subclassable by providing accessors and/or making members protected instead of private. If these classes should not be subclassed, they should be declared as final classes. Thoughts? Opinions? Comments? --- Jim Kellerman, Senior Engineer; Powerset No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.19.5/1228 - Release Date: 1/16/2008 9:01 AM
[jira] Updated: (HADOOP-2600) [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before)
[ https://issues.apache.org/jira/browse/HADOOP-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2600: -- Status: Patch Available (was: Open) Trying Hudson > [hbase] Performance: HStore.getRowKeyAtOrBefore should use > MapFile.Reader#getClosest (before) > - > > Key: HADOOP-2600 > URL: https://issues.apache.org/jira/browse/HADOOP-2600 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2600.patch > > > Once HADOOP-2532 is committed, we should make use of the new MapFile method > to improve the performance of getClosestRow. This is an important > improvement, because this method is used to look up region locations, a very > critical part of HBase. Currently, the needed behavior is simulated by doing > a linear scan of the MapFile. I think that 2532 will allow us to replace this > with an index based lookup, which should make it much faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2600) [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before)
[ https://issues.apache.org/jira/browse/HADOOP-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2600: -- Attachment: 2600.patch This patch updates the code to do binary search instead of linear search. It makes my test suite pass in 18 minutes instead of 22. Sweet! > [hbase] Performance: HStore.getRowKeyAtOrBefore should use > MapFile.Reader#getClosest (before) > - > > Key: HADOOP-2600 > URL: https://issues.apache.org/jira/browse/HADOOP-2600 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2600.patch > > > Once HADOOP-2532 is committed, we should make use of the new MapFile method > to improve the performance of getClosestRow. This is an important > improvement, because this method is used to look up region locations, a very > critical part of HBase. Currently, the needed behavior is simulated by doing > a linear scan of the MapFile. I think that 2532 will allow us to replace this > with an index based lookup, which should make it much faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559322#action_12559322 ] Bryan Duxbury commented on HADOOP-2604: --- It would also be nice to get a row count quickly from a MapFile, or at least check if the MapFile was empty. Can't do that today. > [hbase] Create an HBase-specific MapFile implementation > --- > > Key: HADOOP-2604 > URL: https://issues.apache.org/jira/browse/HADOOP-2604 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Priority: Minor > > Today, HBase uses the Hadoop MapFile class to store data persistently to > disk. This is convenient, as it's already done (and maintained by other > people :). However, it's beginning to look like there might be possible > performance benefits to be had from doing an HBase-specific implementation of > MapFile that incorporated some precise features. > This issue should serve as a place to track discussion about what features > might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Attachment: 2493-v3.patch Here's the forgotten test. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson >Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493-v2.patch, 2493-v3.patch, 2493.patch, > regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Open (was: Patch Available) Forgot to include my new TestSplit2 in the patch > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493-v2.patch, 2493-v3.patch, 2493.patch, > regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Patch Available (was: Open) Trying Hudson. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493-v2.patch, 2493-v3.patch, 2493.patch, > regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Attachment: 2493-v2.patch I've enacted the needed changes in this patch. Tests pass locally. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson >Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493-v2.patch, 2493.patch, regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Patch Available (was: Open) Back to Hudson. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493-v2.patch, 2493.patch, regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Open (was: Patch Available) Despite the fact that the tests pass, stack is right about the busted logic. I will fix that. I will also take the opportunity to convert that method to use the new MapFile.Reader getClosest (before) method to make the lookup more efficient. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493.patch, regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559203#action_12559203 ] Bryan Duxbury commented on HADOOP-2604: --- I think it would make sense for us to maintain both a bloom filter and an index on row keys. That way, you can check the filter first to decide if you should check the index. Even if there's been deletions in the region that damage the filter, the index will still answer your question pretty quickly. We can maintain (re-create) the filter during compactions. I think we would see huge gains from having an always-on bloom filter, especially for sparser row spaces. > [hbase] Create an HBase-specific MapFile implementation > --- > > Key: HADOOP-2604 > URL: https://issues.apache.org/jira/browse/HADOOP-2604 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Priority: Minor > > Today, HBase uses the Hadoop MapFile class to store data persistently to > disk. This is convenient, as it's already done (and maintained by other > people :). However, it's beginning to look like there might be possible > performance benefits to be had from doing an HBase-specific implementation of > MapFile that incorporated some precise features. > This issue should serve as a place to track discussion about what features > might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Status: Patch Available (was: In Progress) Trying hudson. > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2512-v2.patch, 2513.patch > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Attachment: 2512-v2.patch This patch passes all the tests locally. > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2512-v2.patch, 2513.patch > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2592) [hbase] Scanning, a region can let out a row that its not supposed to have
[ https://issues.apache.org/jira/browse/HADOOP-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2592: -- Fix Version/s: 0.16.0 Priority: Critical (was: Minor) This is a pretty critical issue, because it can result in getting an inconsistent answer from a scan. > [hbase] Scanning, a region can let out a row that its not supposed to have > -- > > Key: HADOOP-2592 > URL: https://issues.apache.org/jira/browse/HADOOP-2592 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: stack >Priority: Critical > Fix For: 0.16.0 > > Attachments: test3.patch, tti.patch > > > Bryan's lazy cache patch was provoking a strange issue in TestTableIndex when > flush size was small -- 64k instead of 1MB -- where a scan was returning > random rows twice, w/ the second incidence coming out of a region whose range > shouldn't even contain the row. > After study, HalfMapFiles serving the top half of a parent region down in the > daughter, can have rows in excess of the region endkey; scanning, the call to > next will just return these rows w/ consideration of whether beyond region > endkey. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2592) [hbase] Scanning, a region can let out a row that its not supposed to have
[ https://issues.apache.org/jira/browse/HADOOP-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2592: - Assignee: stack > [hbase] Scanning, a region can let out a row that its not supposed to have > -- > > Key: HADOOP-2592 > URL: https://issues.apache.org/jira/browse/HADOOP-2592 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.16.0 > > Attachments: test3.patch, tti.patch > > > Bryan's lazy cache patch was provoking a strange issue in TestTableIndex when > flush size was small -- 64k instead of 1MB -- where a scan was returning > random rows twice, w/ the second incidence coming out of a region whose range > shouldn't even contain the row. > After study, HalfMapFiles serving the top half of a parent region down in the > daughter, can have rows in excess of the region endkey; scanning, the call to > next will just return these rows w/ consideration of whether beyond region > endkey. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2611) [hbase] Create a RowMutation class in the public API
[
https://issues.apache.org/jira/browse/HADOOP-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559143#action_12559143
]
Bryan Duxbury commented on HADOOP-2611:
---
Also note, this change would allow multiple "concurrent" updates to go on
against the same HTable instance at once, as the changes would be tracked in
the RowMutation instance, rather than internal to the HTable.
> [hbase] Create a RowMutation class in the public API
>
>
> Key: HADOOP-2611
> URL: https://issues.apache.org/jira/browse/HADOOP-2611
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
>Reporter: Bryan Duxbury
>Priority: Minor
>
> Today, when you want to interact with a row in HBase, you start an update,
> make changes, and then commit the lock. This is fine for very simple
> applications. However, when you try to do things like support table
> operations as part of a MapReduce job, it becomes more difficult to support.
> I propose that we create a new class, RowMutation (a la the Bigtable paper),
> which encapsulates a group of actions on a row, and make this available to
> API consumers. It might look something like:
> {code}
> RowMutation r = table.getMutation(row_key);
> r.setTimestamp();
> r.put(new Text("colfam1:name", value));
> r.delete(new Text("colfam2:deleted"));
> table.commit(r);
> {code}
> This syntax would supercede the existing startUpdate/commit format, which
> could be deprecated and mapped to a RowMutation behind the scenes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2611) [hbase] Create a RowMutation class in the public API
[hbase] Create a RowMutation class in the public API
Key: HADOOP-2611
URL: https://issues.apache.org/jira/browse/HADOOP-2611
Project: Hadoop
Issue Type: New Feature
Components: contrib/hbase
Reporter: Bryan Duxbury
Priority: Minor
Today, when you want to interact with a row in HBase, you start an update, make
changes, and then commit the lock. This is fine for very simple applications.
However, when you try to do things like support table operations as part of a
MapReduce job, it becomes more difficult to support.
I propose that we create a new class, RowMutation (a la the Bigtable paper),
which encapsulates a group of actions on a row, and make this available to API
consumers. It might look something like:
{code}
RowMutation r = table.getMutation(row_key);
r.setTimestamp();
r.put(new Text("colfam1:name", value));
r.delete(new Text("colfam2:deleted"));
table.commit(r);
{code}
This syntax would supercede the existing startUpdate/commit format, which could
be deprecated and mapped to a RowMutation behind the scenes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Attachment: 2513.patch This patch includes some new tests in TestGet2 to highlight this problem and fixes in HStore to take care of the issues. > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2513.patch > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2513: - Assignee: Bryan Duxbury > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury > Fix For: 0.16.0 > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-2513 started by Bryan Duxbury. > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury > Fix For: 0.16.0 > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Status: Patch Available (was: In Progress) Passes locally, sending to Hudson. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493.patch, regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Attachment: 2493.patch This patch adds a new test, TestSplit2, that makes sure a region of all the same row key will not be split. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: 2493.patch, regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2493: - Assignee: Bryan Duxbury > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-2493 started by Bryan Duxbury. > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson > Assignee: Bryan Duxbury >Priority: Critical > Fix For: 0.16.0 > > Attachments: regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[ https://issues.apache.org/jira/browse/HADOOP-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558846#action_12558846 ] Bryan Duxbury commented on HADOOP-2604: --- Here's some of the ideas we're tossing around as a starter: * Exclude column family name from the file: Currently we store HStoreKeys, which are serialized to contain row, qualified cell name, and timestamp. However, seeing as how a given MapFile only ever belongs to one column family it's very wasteful to store the same column family name over and over again. In a custom implementation, we wouldn't have to save that data. * Separate indices for rows from qualified name and timestamp: Currently, the index in MapFiles is over all records, so the same row can appear in the index more than one time (differentiated by column name/timestamp). If the index just contained row keys, then we could store each row key exactly once, which would point to a record group of qualified names and timestamps (and values of course). Within the record group, there could be another separate small index on qualified name. This would again reduce the size of data stored, size of indices, and make it easier to do things like split regions lexically instead of skewed by cell count. * Use random rather than streaming reads: There is some indication that the existing MapFile implementation is optimized for streaming access; HBase supports random reads, which are therefore not efficient under MapFile. It would make sense for us to design our new implementation in such a way that it would be very cheap to do random access. > [hbase] Create an HBase-specific MapFile implementation > --- > > Key: HADOOP-2604 > URL: https://issues.apache.org/jira/browse/HADOOP-2604 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Priority: Minor > > Today, HBase uses the Hadoop MapFile class to store data persistently to > disk. This is convenient, as it's already done (and maintained by other > people :). However, it's beginning to look like there might be possible > performance benefits to be had from doing an HBase-specific implementation of > MapFile that incorporated some precise features. > This issue should serve as a place to track discussion about what features > might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2604) [hbase] Create an HBase-specific MapFile implementation
[hbase] Create an HBase-specific MapFile implementation --- Key: HADOOP-2604 URL: https://issues.apache.org/jira/browse/HADOOP-2604 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HBase uses the Hadoop MapFile class to store data persistently to disk. This is convenient, as it's already done (and maintained by other people :). However, it's beginning to look like there might be possible performance benefits to be had from doing an HBase-specific implementation of MapFile that incorporated some precise features. This issue should serve as a place to track discussion about what features might be included in such an implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2579) initializing a new HTable object against a nonexistent table throws a NoServerForRegionException instead of a TableNotFoundException when a different table has been creat
[
https://issues.apache.org/jira/browse/HADOOP-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Duxbury updated HADOOP-2579:
--
Status: Patch Available (was: Open)
Tests pass locally, submitting to Hudson.
> initializing a new HTable object against a nonexistent table throws a
> NoServerForRegionException instead of a TableNotFoundException when a
> different table has been created previously
> ---
>
> Key: HADOOP-2579
> URL: https://issues.apache.org/jira/browse/HADOOP-2579
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
>Reporter: Peter Dolan
> Attachments: 2579.patch
>
>
> When a table has been created, initializing a new HTable object for that
> table works properly, as expected, but initializing a new HTable against a
> nonexistent table in the same context (a table exists), a
> NoServerForRegionException is thrown instead of TableNotFoundException, as
> would be expected.
> If there haven't been any tables created, initializing a new HTable object
> against 'nosuchTable' throws TableNotFoundException, as expected.
> Running the following TestCase exposes the issue on my machine.
> {code:title=TestHTable.java|borderStyle=solid}
> package org.apache.hadoop.hbase;
> import java.io.IOException;
> import org.apache.hadoop.io.Text;
> /**
> * Tests HTable
> */
> public class TestHTable extends HBaseClusterTestCase implements HConstants {
> public void testTableNotFoundExceptionWithoutAnyTables() {
> try {
> new HTable(conf, new Text("notATable"));
> fail("Should have thrown a TableNotFoundException");
> } catch (TableNotFoundException e) {
> // expected
> } catch (IOException e) {
> e.printStackTrace();
> fail("Should have thrown a TableNotFoundException instead of a " +
> e.getClass());
> }
> }
>
> public void testTableNotFoundExceptionWithATable() {
> try {
> HColumnDescriptor column =
> new HColumnDescriptor(COLUMN_FAMILY.toString());
> HBaseAdmin admin = new HBaseAdmin(conf);
> HTableDescriptor testTableADesc =
> new HTableDescriptor("table");
> testTableADesc.addFamily(column);
> admin.createTable(testTableADesc);
> // This should throw a TableNotFoundException, it has not been created
> new HTable(conf, new Text("notATable"));
>
> fail("Should have thrown a TableNotFoundException");
> } catch (TableNotFoundException e) {
> // expected
> } catch (IOException e) {
> e.printStackTrace();
> fail("Should have thrown a TableNotFoundException instead of a " +
> e.getClass());
> }
> }
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2579) initializing a new HTable object against a nonexistent table throws a NoServerForRegionException instead of a TableNotFoundException when a different table has been creat
[
https://issues.apache.org/jira/browse/HADOOP-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Duxbury updated HADOOP-2579:
--
Attachment: 2579.patch
This patch includes Peter's new test cases and some fixes to HTable,
HConnectionManager, HRegionServer, HRegion, and HStore to correctly handle the
case when there's no tables at all in .META.
> initializing a new HTable object against a nonexistent table throws a
> NoServerForRegionException instead of a TableNotFoundException when a
> different table has been created previously
> ---
>
> Key: HADOOP-2579
> URL: https://issues.apache.org/jira/browse/HADOOP-2579
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
>Reporter: Peter Dolan
> Attachments: 2579.patch
>
>
> When a table has been created, initializing a new HTable object for that
> table works properly, as expected, but initializing a new HTable against a
> nonexistent table in the same context (a table exists), a
> NoServerForRegionException is thrown instead of TableNotFoundException, as
> would be expected.
> If there haven't been any tables created, initializing a new HTable object
> against 'nosuchTable' throws TableNotFoundException, as expected.
> Running the following TestCase exposes the issue on my machine.
> {code:title=TestHTable.java|borderStyle=solid}
> package org.apache.hadoop.hbase;
> import java.io.IOException;
> import org.apache.hadoop.io.Text;
> /**
> * Tests HTable
> */
> public class TestHTable extends HBaseClusterTestCase implements HConstants {
> public void testTableNotFoundExceptionWithoutAnyTables() {
> try {
> new HTable(conf, new Text("notATable"));
> fail("Should have thrown a TableNotFoundException");
> } catch (TableNotFoundException e) {
> // expected
> } catch (IOException e) {
> e.printStackTrace();
> fail("Should have thrown a TableNotFoundException instead of a " +
> e.getClass());
> }
> }
>
> public void testTableNotFoundExceptionWithATable() {
> try {
> HColumnDescriptor column =
> new HColumnDescriptor(COLUMN_FAMILY.toString());
> HBaseAdmin admin = new HBaseAdmin(conf);
> HTableDescriptor testTableADesc =
> new HTableDescriptor("table");
> testTableADesc.addFamily(column);
> admin.createTable(testTableADesc);
> // This should throw a TableNotFoundException, it has not been created
> new HTable(conf, new Text("notATable"));
>
> fail("Should have thrown a TableNotFoundException");
> } catch (TableNotFoundException e) {
> // expected
> } catch (IOException e) {
> e.printStackTrace();
> fail("Should have thrown a TableNotFoundException instead of a " +
> e.getClass());
> }
> }
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-2599 started by Bryan Duxbury. > [hbase] Some minor improvements to changes in HADOOP-2443 > - > > Key: HADOOP-2599 > URL: https://issues.apache.org/jira/browse/HADOOP-2599 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2599.patch > > > Stack made a number of suggestions for improvements in the code changes from > HADOOP-2443: > Here in HConnectionManager: > + scannerId = server.openScanner( > +metaLocation.getRegionInfo().getRegionName(), > +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, > +null); > ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd > work too - right?). > Here: > + } while (startRow.compareTo(EMPTY_START_ROW) != 0); > .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't > you think? > This 999... 'trick' should be done as a method - in HTableInfo? - since you > do it twice: > +Text tableKey = new Text(tableName.toString() + ",,99"); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2599: -- Attachment: 2599.patch Here's a patch that implements the first two suggestions. > [hbase] Some minor improvements to changes in HADOOP-2443 > - > > Key: HADOOP-2599 > URL: https://issues.apache.org/jira/browse/HADOOP-2599 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury >Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2599.patch > > > Stack made a number of suggestions for improvements in the code changes from > HADOOP-2443: > Here in HConnectionManager: > + scannerId = server.openScanner( > +metaLocation.getRegionInfo().getRegionName(), > +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, > +null); > ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd > work too - right?). > Here: > + } while (startRow.compareTo(EMPTY_START_ROW) != 0); > .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't > you think? > This 999... 'trick' should be done as a method - in HTableInfo? - since you > do it twice: > +Text tableKey = new Text(tableName.toString() + ",,99"); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2600) [hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before)
[hbase] Performance: HStore.getRowKeyAtOrBefore should use MapFile.Reader#getClosest (before) - Key: HADOOP-2600 URL: https://issues.apache.org/jira/browse/HADOOP-2600 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor Once HADOOP-2532 is committed, we should make use of the new MapFile method to improve the performance of getClosestRow. This is an important improvement, because this method is used to look up region locations, a very critical part of HBase. Currently, the needed behavior is simulated by doing a linear scan of the MapFile. I think that 2532 will allow us to replace this with an index based lookup, which should make it much faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558708#action_12558708 ] Bryan Duxbury commented on HADOOP-2599: --- @stack: Regarding the third suggestion, are you indicating that we need a new class to hold a method that can append an unreasonably large timestamp onto a table name? Does the method seem complex enough to warrant that? If we create an HTableInfo class, what else would go in it? > [hbase] Some minor improvements to changes in HADOOP-2443 > - > > Key: HADOOP-2599 > URL: https://issues.apache.org/jira/browse/HADOOP-2599 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > > Stack made a number of suggestions for improvements in the code changes from > HADOOP-2443: > Here in HConnectionManager: > + scannerId = server.openScanner( > +metaLocation.getRegionInfo().getRegionName(), > +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, > +null); > ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd > work too - right?). > Here: > + } while (startRow.compareTo(EMPTY_START_ROW) != 0); > .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't > you think? > This 999... 'trick' should be done as a method - in HTableInfo? - since you > do it twice: > +Text tableKey = new Text(tableName.toString() + ",,99"); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[ https://issues.apache.org/jira/browse/HADOOP-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2599: -- Component/s: contrib/hbase Assignee: Bryan Duxbury Description: Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ",,99"); was: Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ",,99"); [ Show » ] stack - 13/Jan/08 10:54 PM Committed. Had some minor questions - see below - but it ain't enough to hold up patch and I'm afraid patch would rot if left hang (It touches lots of classes). Resolving. Thanks for the patch Bryan. Here's the couple of items: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ",,99"); > [hbase] Some minor improvements to changes in HADOOP-2443 > - > > Key: HADOOP-2599 > URL: https://issues.apache.org/jira/browse/HADOOP-2599 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: Bryan Duxbury >Priority: Minor > > Stack made a number of suggestions for improvements in the code changes from > HADOOP-2443: > Here in HConnectionManager: > + scannerId = server.openScanner( > +metaLocation.getRegionInfo().getRegionName(), > +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, > +null); > ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd > work too - right?). > Here: > + } while (startRow.compareTo(EMPTY_START_ROW) != 0); > .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't > you think? > This 999... 'trick' should be done as a method - in HTableInfo? - since you > do it twice: > +Text tableKey = new Text(tableName.toString() + ",,99"); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury resolved HADOOP-2443. --- Resolution: Fixed Resolving issue. I have opened new issues for dealing with the problems mentioned in comments above. See HADOOP-2598 and HADOOP-2599. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, > 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2599) [hbase] Some minor improvements to changes in HADOOP-2443
[hbase] Some minor improvements to changes in HADOOP-2443 - Key: HADOOP-2599 URL: https://issues.apache.org/jira/browse/HADOOP-2599 Project: Hadoop Issue Type: Improvement Reporter: Bryan Duxbury Priority: Minor Stack made a number of suggestions for improvements in the code changes from HADOOP-2443: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ",,99"); [ Show » ] stack - 13/Jan/08 10:54 PM Committed. Had some minor questions - see below - but it ain't enough to hold up patch and I'm afraid patch would rot if left hang (It touches lots of classes). Resolving. Thanks for the patch Bryan. Here's the couple of items: Here in HConnectionManager: + scannerId = server.openScanner( +metaLocation.getRegionInfo().getRegionName(), +COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP, +null); ... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too - right?). Here: + } while (startRow.compareTo(EMPTY_START_ROW) != 0); .. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think? This 999... 'trick' should be done as a method - in HTableInfo? - since you do it twice: +Text tableKey = new Text(tableName.toString() + ",,99"); -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[ https://issues.apache.org/jira/browse/HADOOP-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2598: -- Status: Patch Available (was: Open) Trying hudson. > [hbase] Remove chatty debug logging from 2443 patch > --- > > Key: HADOOP-2598 > URL: https://issues.apache.org/jira/browse/HADOOP-2598 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Fix For: 0.16.0 > > Attachments: 2598.patch > > > There was some leftover debug logging in HRegionServer in the patch for 2443. > Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[ https://issues.apache.org/jira/browse/HADOOP-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury reassigned HADOOP-2598: - Assignee: Bryan Duxbury > [hbase] Remove chatty debug logging from 2443 patch > --- > > Key: HADOOP-2598 > URL: https://issues.apache.org/jira/browse/HADOOP-2598 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Fix For: 0.16.0 > > Attachments: 2598.patch > > > There was some leftover debug logging in HRegionServer in the patch for 2443. > Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[ https://issues.apache.org/jira/browse/HADOOP-2598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2598: -- Attachment: 2598.patch Actually, it looks like the message was in HConnectionManager$TableServers for checking the region location cache. This patch removes that as well as a commented logging line in HRegionServer. > [hbase] Remove chatty debug logging from 2443 patch > --- > > Key: HADOOP-2598 > URL: https://issues.apache.org/jira/browse/HADOOP-2598 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: Bryan Duxbury >Priority: Minor > Fix For: 0.16.0 > > Attachments: 2598.patch > > > There was some leftover debug logging in HRegionServer in the patch for 2443. > Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2598) [hbase] Remove chatty debug logging from 2443 patch
[hbase] Remove chatty debug logging from 2443 patch --- Key: HADOOP-2598 URL: https://issues.apache.org/jira/browse/HADOOP-2598 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Fix For: 0.16.0 There was some leftover debug logging in HRegionServer in the patch for 2443. Remove. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2597) [hbase] Performance - add a block cache
[ https://issues.apache.org/jira/browse/HADOOP-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558634#action_12558634 ] Bryan Duxbury commented on HADOOP-2597: --- I like this idea. We could implement it with a wrapper around MapFile, something like CachingMapFile. > [hbase] Performance - add a block cache > --- > > Key: HADOOP-2597 > URL: https://issues.apache.org/jira/browse/HADOOP-2597 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Tom White > > A block cache would cache fixed size blocks (default 64k) of data read from > HDFS by the MapFile. It would help read performance for data close to > recently read data (see Bigtable paper, section 6). It would be configurable > on a per-column family basis. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v10.patch Ok, here's another go. The TestGet2 failure I had previously was actually correct, as I wasn't shutting down the test correctly. I fixed that, and in the process, merged in the changes from other recent patches. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, > 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) Try the latest again. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, > 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2584) Web UI displays an IOException instead of the Tables
[ https://issues.apache.org/jira/browse/HADOOP-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2584: -- Priority: Trivial (was: Major) This issue appears to be a cosmetic ui interface from what I can tell. I'm reducing the priority to match this problem. > Web UI displays an IOException instead of the Tables > > > Key: HADOOP-2584 > URL: https://issues.apache.org/jira/browse/HADOOP-2584 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Affects Versions: 0.15.2 >Reporter: Lars George >Priority: Trivial > > For me after every second restart I get an error when loading the Hbase UI. > Here the page: > > Master: 192.168.105.11:6 > > HQL, Local logs, Thread Dump, Log Level > > > __ > > > > Master Attributes > > > Attribute Name > Value Description > >Filesystem lv1-xen-pdc-2.worldlingo.com:9000 Filesystem hbase is > running on >Hbase Root Directory /hbaseLocation of hbase > home directory > > > Online META Regions > > > > Name Server > >-ROOT-192.168.105.31:60020 > >.META.,,1 192.168.105.39:60020 > > > > Tables > > error msg : java.io.IOException: java.io.IOException: HStoreScanner failed > construction at > org.apache.hadoop.hbase.HStore$StoreFileScanner.(HStore.java:1879) at > org.apache.hadoop.hbase.HStore$HStoreScanner.(HStore.java:2000) at > org.apache.hadoop.hbase.HStore.getScanner(HStore.java:1822) at > org.apache.hadoop.hbase.HRegion$HScanner.(HRegion.java:1543) at > org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1118) at > org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1465) > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at >java.lang.reflect.Method.invoke(Method.java:585) at > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401) at >org.apache.hadoop.ipc.Server$Handler.run(Server.java:892) Caused by: > org.apache.hadoop.ipc.RemoteException: >java.io.IOException: File does not exist: > /hbase/hregion_1028785192/info/mapfiles/6628785818889695133/data at > >org.apache.hadoop.dfs.FSDirectory.getFileInfo(FSDirectory.java:489) at > >org.apache.hadoop.dfs.FSNamesystem.getFileInfo(FSName
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) Tests failed by TestGet2 timing out. No idea what happened there, so I'll resubmit. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2542) Opening then closing a scanner delays the effects of a delete
[ https://issues.apache.org/jira/browse/HADOOP-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2542: -- Status: Open (was: Patch Available) Patch just contains a unit test that is intended to fail. Only need to "submit" a patch once it's ready to be tested for committing. > Opening then closing a scanner delays the effects of a delete > - > > Key: HADOOP-2542 > URL: https://issues.apache.org/jira/browse/HADOOP-2542 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Clint Morgan > Attachments: HADOOP-2542.patch > > > I stumbled across the following odd behavior: When I open, then close a > scanner on > a table, then delete a row in the table, the delete does not work > (i.e., I can still get the row). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2440) [hbase] Provide a HBase checker and repair tool similar to fsck
[ https://issues.apache.org/jira/browse/HADOOP-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558447#action_12558447 ] Bryan Duxbury commented on HADOOP-2440: --- Should we adjust the fix version out to 0.17 if the main issue hasn't been addressed yet? > [hbase] Provide a HBase checker and repair tool similar to fsck > --- > > Key: HADOOP-2440 > URL: https://issues.apache.org/jira/browse/HADOOP-2440 > Project: Hadoop > Issue Type: New Feature > Components: contrib/hbase >Affects Versions: 0.16.0 >Reporter: Jim Kellerman >Assignee: Jim Kellerman > Fix For: 0.16.0 > > Attachments: patch.txt > > > We need a tool to verify (and repair) HBase much like fsck -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2591) [HQL] Addition of start command for execute the *.hql query file
[
https://issues.apache.org/jira/browse/HADOOP-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558446#action_12558446
]
Bryan Duxbury commented on HADOOP-2591:
---
Feature freeze seems to be this Friday. Is this issue going to be completed by
then? If not, please adjust the fix version.
> [HQL] Addition of start command for execute the *.hql query file
>
>
> Key: HADOOP-2591
> URL: https://issues.apache.org/jira/browse/HADOOP-2591
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
>Affects Versions: 0.16.0
>Reporter: Edward Yoon
>Assignee: Edward Yoon
> Fix For: 0.16.0
>
>
> {code}
> $ cat queries.hql
> CREATE TABLE webtable (
> contents in_memory max_versions=10 compression=block,
> anchor max_length=256 bloomfilter=counting_bloomfilter
> vector_size=100 num_hash=4);
> );
> INSERT INTO webtable (contents, anchor:Hadoop)
> VALUES ('html content', 'http://hadoop.apache.org')
> WHERE row='http://www.hadoop.co.kr';
> ...
>
> hql> start ./queries.hql;
> ...
> hql> desc webtable;
>
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) Trying Hudson. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v9.patch Merged in Jim's patch in this version. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: Open) Failure was in TestTableIndex in a place I'd never seen problems before. It was down in Flusher. I'm going to resubmit it and see if it passes again. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Open (was: Patch Available) > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Status: Patch Available (was: In Progress) Trying on hudson. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v8.patch Fixed javadoc warning. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch, 2443-v8.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v7.patch Another version, more cleaned up. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch, 2443-v7.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2555) Refactor the HTable#get and HTable#getRow methods to avoid repetition of retry-on-failure logic
[
https://issues.apache.org/jira/browse/HADOOP-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558191#action_12558191
]
Bryan Duxbury commented on HADOOP-2555:
---
Generally, I love this patch. It'll nicely reduce the amount of copy-paste we
have.
I'd like to maybe take it one step further, though. Instead of the existing
callServerWithRetries, how about something like:
{code}
protected abstract class ServerCallable implements Callable {
HRegionLocation location;
HRegionInterface server;
Text row;
protected ServerCallable(Text row){
this.row = row;
}
void instantiateServer(boolean reload) throws IOException {
if (reload) {
tableServers = connection.reloadTableServers(tableName);
}
location = getRegionLocation(row);
server = connection.getHRegionConnection(location.getServerAddress());
}
}
protected T getRegionServerWithRetries(ServerCallable callable)
throws IOException, UnexpectedCallableException {
for(int tries = 0; tries < numRetries; tries++) {
try {
callable.instantiateServer(tries == 0);
return callable.call();
} catch (IOException e) {
if (e instanceof RemoteException) {
e = RemoteExceptionHandler.decodeRemoteException((RemoteException) e);
}
if (tries == numRetries - 1) {
throw e;
}
if (LOG.isDebugEnabled()) {
LOG.debug("reloading table servers because: " + e.getMessage());
}
} catch (Exception e) {
throw new UnexpectedCallableException(e);
}
try {
Thread.sleep(pause);
} catch (InterruptedException e) {
// continue
}
}
return null;
}
{code}
which takes us from
{code}
value = this.callServerWithRetries(new Callable() {
public MapWritable call() throws IOException {
HRegionLocation r = getRegionLocation(row);
HRegionInterface server =
connection.getHRegionConnection(r.getServerAddress());
return server.getRow(r.getRegionInfo().getRegionName(), row, ts);
}
});
{code}
to
{code}
value = this.callServerWithRetries(new ServerCallable(row) {
public MapWritable call() throws IOException {
return server.getRow(location.getRegionInfo().getRegionName(), row, ts);
}
});
{code}
This would save a few lines of code inside each internal block, move a little
more logic into the helper method, and generally jive better with the way my
HADOOP-2443 patch is going to need to work in the near future.
Comments?
> Refactor the HTable#get and HTable#getRow methods to avoid repetition of
> retry-on-failure logic
> ---
>
> Key: HADOOP-2555
> URL: https://issues.apache.org/jira/browse/HADOOP-2555
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
>Reporter: Peter Dolan
>Priority: Minor
> Attachments: hadoop-2555.patch
>
>
> The following code is repeated in every one of HTable#get and HTable#getRow
> methods:
> {code:title=HTable.java|borderStyle=solid}
> MapWritable value = null;
> for (int tries = 0; tries < numRetries; tries++) {
> HRegionLocation r = getRegionLocation(row);
> HRegionInterface server =
> connection.getHRegionConnection(r.getServerAddress());
>
> try {
> value = server.getRow(r.getRegionInfo().getRegionName(), row, ts);
> // This is the only line of code that changes significantly between methods
> break;
>
> } catch (IOException e) {
> if (e instanceof RemoteException) {
> e = RemoteExceptionHandler.decodeRemoteException((RemoteException)
> e);
> }
> if (tries == numRetries - 1) {
> // No more tries
> throw e;
> }
> if (LOG.isDebugEnabled()) {
> LOG.debug("reloading table servers because: " + e.getMessage());
> }
> tableServers = connection.reloadTableServers(tableName);
> }
> try {
> Thread.sleep(this.pause);
>
> } catch (InterruptedException x) {
> // continue
> }
> }
> {code}
> This should be factored out into a protected method that handles
> retry-on-failure logic to facilitate more robust testing and the development
> of new API methods.
> Proposed modification:
> // Execute the provided Callable against the server
> protected callServerWithRetries(Callable callable) throws
> RemoteException;
> The above code could then be reduced to:
> {cod
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v6.patch All the tests pass locally at this point, big improvement over the last version. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, > 2443-v6.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2500) [HBase] Unreadable region kills region servers
[ https://issues.apache.org/jira/browse/HADOOP-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558036#action_12558036 ] Bryan Duxbury commented on HADOOP-2500: --- So, we should: * Change the "no good" message to something a tad more descriptive, like "assignment of region is invalid" * Enumerate the known ways that a RegionServer can fail to serve a region, trap those problems, and figure out what responses we'd like to give to those events > [HBase] Unreadable region kills region servers > -- > > Key: HADOOP-2500 > URL: https://issues.apache.org/jira/browse/HADOOP-2500 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Environment: CentOS 5 >Reporter: Chris Kline >Priority: Critical > > Backgound: The name node (also a DataNode and RegionServer) in our cluster > ran out of disk space. I created some space, restarted HDFS and fsck > reported corruption with an HBase file. I cleared up that corruption and > restarted HBase. I was still unable to read anything from HBase even though > HSFS was now healthy. > The following was gather from the log files. When HMaster starts up, it > finds a region that is no good (Key: 17_125736271): > 2007-12-24 09:07:14,342 DEBUG org.apache.hadoop.hbase.HMaster: Current > assignment of spider_pages,17_125736271,1198286140018 is no good > HMaster then assigns this region to RegionServer X.60: > 2007-12-24 09:07:17,126 INFO org.apache.hadoop.hbase.HMaster: assigning > region spider_pages,17_125736271,1198286140018 to server 10.100.11.60:60020 > 2007-12-24 09:07:20,152 DEBUG org.apache.hadoop.hbase.HMaster: Received > MSG_REPORT_PROCESS_OPEN : spider_pages,17_125736271,1198286140018 from > 10.100.11.60:60020 > The RegionServer has trouble reading that region (from the RegionServer log > on X.60); Note that the worker thread exits > 2007-12-24 09:07:22,611 DEBUG org.apache.hadoop.hbase.HStore: starting > spider_pages,17_125736271,1198286140018/meta (2062710340/meta with > reconstruction log: (/data/hbase1/hregion_2062710340/oldlogfile.log > 2007-12-24 09:07:22,620 DEBUG org.apache.hadoop.hbase.HStore: maximum > sequence id for hstore spider_pages,17_125736271,1198286140018/meta > (2062710340/meta) is 4549496 > 2007-12-24 09:07:22,622 ERROR org.apache.hadoop.hbase.HRegionServer: error > opening region spider_pages,17_125736271,1198286140018 > java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at java.io.DataInputStream.readFully(DataInputStream.java:152) > at > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1383) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1360) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1349) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1344) > at org.apache.hadoop.hbase.HStore.doReconstructionLog(HStore.java:697) > at org.apache.hadoop.hbase.HStore.(HStore.java:632) > at org.apache.hadoop.hbase.HRegion.(HRegion.java:288) > at > org.apache.hadoop.hbase.HRegionServer.openRegion(HRegionServer.java:1211) > at > org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:1162) > at java.lang.Thread.run(Thread.java:619) > 2007-12-24 09:07:22,623 FATAL org.apache.hadoop.hbase.HRegionServer: > Unhandled exception > java.lang.NullPointerException > at > org.apache.hadoop.hbase.HRegionServer.reportClose(HRegionServer.java:1095) > at > org.apache.hadoop.hbase.HRegionServer.openRegion(HRegionServer.java:1217) > at > org.apache.hadoop.hbase.HRegionServer$Worker.run(HRegionServer.java:1162) > at java.lang.Thread.run(Thread.java:619) > 2007-12-24 09:07:22,623 INFO org.apache.hadoop.hbase.HRegionServer: worker > thread exiting > The HMaster then tries to assign the same region to X.60 again and fails. > The HMaster tries to assign the region to X.31 with the same result (X.31 > worker thread exits). > The file it is complaining about, > /data/hbase1/hregion_2062710340/oldlogfile.log, is a zero-length file in > HDFS. After deleting that file and restarting HBase, HBase appears to be > back to normal. > One thing I can't figure out is that the HMaster log show several entries > after the worker thread on X.60 has exited suggesting that the RegionServer > is talking with HMaster: > 2007-12-24 09:08:23,349 DEBUG org.apache.hadoop.hbase.HMaster: Received > MS
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v5.patch The latest version of the patch removes getTableServers and related methods entirely and switches HTable$ClientScanner to use getRegionLocation. Unfortunately, I now have TestTableIndex and TestTableMapReduce failing all the time. The error seem to be related to scanners acting strangely with rows in the wrong regions, but I can't be sure of what's going on yet. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2496) Snapshot of table
[ https://issues.apache.org/jira/browse/HADOOP-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2496: -- Fix Version/s: (was: 0.16.0) 0.17.0 Priority: Minor (was: Major) Adjusting the priority down to Minor, as this is new functionality. Also setting Fix Version to 0.17, as we have a lot of stuff to get done before 0.16 as it is. > Snapshot of table > - > > Key: HADOOP-2496 > URL: https://issues.apache.org/jira/browse/HADOOP-2496 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson >Priority: Minor > Fix For: 0.17.0 > > > Havening an option to take a snapshot of a table would be vary useful in > production. > What I would like to see this option do is do a merge of all the data into > one or more files stored in the same folder on the dfs. This way we could > save data in case of a software bug in hadoop or user code. > The other advantage would be to be able to export a table to multi locations. > Say I had a read_only table that must be online. I could take a snapshot of > it when needed and export it to a separate data center and have it loaded > there and then i would have it online at multi data centers for load > balancing and failover. > I understand that hadoop takes the need out of havening backup to protect > from failed servers, but this does not protect use from software bugs that > might delete or alter data in ways we did not plan. We should have a way we > can roll back a dataset. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2478) [hbase] restructure how HBase lays out files in the file system
[
https://issues.apache.org/jira/browse/HADOOP-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557514#action_12557514
]
Bryan Duxbury commented on HADOOP-2478:
---
If this won't be fixed until 0.17, which is months away, should we apply my
HStore refactor patch in the meantime?
> [hbase] restructure how HBase lays out files in the file system
> ---
>
> Key: HADOOP-2478
> URL: https://issues.apache.org/jira/browse/HADOOP-2478
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
>Affects Versions: 0.17.0
>Reporter: Jim Kellerman
>Assignee: Jim Kellerman
> Fix For: 0.17.0
>
> Attachments: patch.txt
>
>
> Currently HBase has a pretty flat directory structure. For example:
> {code}
> /hbase/hregion_70236052/info
> /hbase/hregion_70236052/info/info/4328260619704027575
> /hbase/hregion_70236052/info/mapfiles/4328260619704027575
> /hbase/hregion_70236052/info/mapfiles/4328260619704027575/data
> /hbase/hregion_70236052/info/mapfiles/4328260619704027575/index
> {code}
> All the region directories are under the root directory, and with encoded
> region names, it is impossible to determine what table a region belongs to.
> This should be restructured to:
> {code}
> /root-directory/table-name/encoded-region-name/column-family/{info,mapfiles}
> {code}
> It will be necessary to provide a migration script from current trunk to the
> new structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2532) Add to MapFile a getClosest that returns key that comes just-before if key not present (Currently does just-after only).
[ https://issues.apache.org/jira/browse/HADOOP-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557439#action_12557439 ] Bryan Duxbury commented on HADOOP-2532: --- +1 This should make my HBase cache patch much more efficient. > Add to MapFile a getClosest that returns key that comes just-before if key > not present (Currently does just-after only). > > > Key: HADOOP-2532 > URL: https://issues.apache.org/jira/browse/HADOOP-2532 > Project: Hadoop > Issue Type: New Feature >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 0.16.0 > > Attachments: getclosestbefore-v2.patch, getclosestbefore-v3.patch, > getclosestbefore.patch > > > The list of regions that make up a table in hbase are effectively kept in a > mapfile. Regions are identified by the first row contained by that region. > To find the region that contains a particular row, we need to be able to > search the mapfile of regions to find the closest matching row that falls > just-before the searched-for key rather than the just-after that is current > mapfile getClosest behavior. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v4.patch All tests are passing locally. This patch includes caching, so performance should be comparable to what you see with the other region location lookup approach. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch, 2443-v4.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Attachment: 2443-v3.patch Not quite done yet - a few tests don't pass, and there is no caching at all right now. > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > Attachments: 2443-v3.patch > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2545) hbase rest server should be started with hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HADOOP-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556805#action_12556805 ] Bryan Duxbury commented on HADOOP-2545: --- Can you suggest what exactly you'd like to see change in the way the REST servlet is started? What advantages do you gain? > hbase rest server should be started with hbase-daemon.sh > > > Key: HADOOP-2545 > URL: https://issues.apache.org/jira/browse/HADOOP-2545 > Project: Hadoop > Issue Type: New Feature > Components: contrib/hbase >Reporter: Michael Bieniosek >Priority: Minor > > Currently, the hbase rest server is started with the hbase script. But it > should be started with the hbase-daemon script, which allows better > configuration options. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2545) hbase rest server should be started with hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HADOOP-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2545: -- Priority: Minor (was: Major) > hbase rest server should be started with hbase-daemon.sh > > > Key: HADOOP-2545 > URL: https://issues.apache.org/jira/browse/HADOOP-2545 > Project: Hadoop > Issue Type: New Feature > Components: contrib/hbase >Reporter: Michael Bieniosek >Priority: Minor > > Currently, the hbase rest server is started with the hbase script. But it > should be started with the hbase-daemon script, which allows better > configuration options. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2493) hbase will split on row when the start and end row is the same cuase data loss
[ https://issues.apache.org/jira/browse/HADOOP-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2493: -- Fix Version/s: 0.16.0 > hbase will split on row when the start and end row is the same cuase data loss > -- > > Key: HADOOP-2493 > URL: https://issues.apache.org/jira/browse/HADOOP-2493 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Reporter: Billy Pearson >Priority: Critical > Fix For: 0.16.0 > > Attachments: regions_shot.JPG > > > While testing hbase splits with my code I was loading a table to become a > inverted index on some links > I was using the anchor text as the row key > and the column parent:child as > url:(siteurl) and the data is the count of the links pointing to the siteurl > with row key anchor text. > but a lot of sites have image links and I use "image" as the anchor text for > my testing code so there is a lot of image links. > I changed the max file size of hbase to 16mb for testing and have been able > to recreate the same error. > When the table get big it splits on the column image as the end key for one > table and the start of the next table later it splits to where the start key > and end key was image for one of the splits. After that it keep spiting the > region with start key as "image" and the end key the same. So I have multi > splits with start key and end key as "image" unless the master keeps track of > the row key and partend:child data on the splits I do not thank all the data > will get returned when querying it. > I have attached a screen shot of my regions i thank there should be some > logic to where if the start and end row key is the same the region does not > split or we need to start keeping track of the start key, column data on the > master of each split so we can know where each row is in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[ https://issues.apache.org/jira/browse/HADOOP-2513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2513: -- Fix Version/s: 0.16.0 > [hbase] HStore#get and HStore#getFull may not return expected values by > timestamp when there is more than one MapFile > - > > Key: HADOOP-2513 > URL: https://issues.apache.org/jira/browse/HADOOP-2513 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Fix For: 0.16.0 > > > Ok, this one is a little tricky. Let's say that you write a row with some > value without a timestamp, thus meaning right now. Then, the memcache gets > flushed out to a MapFile. Then, you write another value to the same row, this > time with a timestamp that is in the past, ie, before the "now" timestamp of > the first put. > Some time later, but before there is a compaction, if you do a get for this > row, and only ask for a single version, you will logically be expecting the > latest version of the cell, which you would assume would be the one written > at "now" time. Instead, you will get the value written into the "past" cell, > because even though it is tagged as having happened in the past, it actually > *was written* after the "now" cell, and thus when #get searches for > satisfying values, it runs into the one most recently written first. > The result of this problem is inconsistent data results. Note that this > problem only ever exists when there's an uncompacted HStore, because during > compaction, these cells will all get sorted into the correct order by > timestamp and such. In a way, this actually makes the problem worse, because > then you could easily get inconsistent results from HBase about the same > (unchanged) row depending on whether there's been a flush/compaction. > The only solution I can think of for this problem at the moment is to scan > all the MapFiles and Memcache for possible results, sort them, and then > select the desired number of versions off of the top. This is unfortunate > because it means you never get the snazzy shortcircuit logic except within a > single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2443: -- Fix Version/s: 0.16.0 > [hbase] Keep lazy cache of regions in client rather than an 'authoritative' > list > > > Key: HADOOP-2443 > URL: https://issues.apache.org/jira/browse/HADOOP-2443 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase > Reporter: stack >Assignee: Bryan Duxbury > Fix For: 0.16.0 > > > Currently, when the client gets a NotServingRegionException -- usually > because its in middle of being split or there has been a regionserver crash > and region is being moved elsewhere -- the client does a complete refresh of > its cache of region locations for a table. > Chatting with Jim about a Paul Saab upload issue from Saturday night, when > tables are big comprised of regions that are splitting fast (because of bulk > upload), its unlikely a client will ever be able to obtain a stable list of > all region locations. Given that any update or scan requires that the list > of all regions be in place before it proceeds, this can get in the way of the > client succeeding when the cluster is under load. > Chatting, we figure that it better the client holds a lazy region cache: on > NSRE, figure out where that region has gone only and update the client-side > cache for that entry only rather than throw out all we know of a table every > time. > Hopefully this will fix the issue PS was experiencing where during intense > upload, he was unable to get/scan/hql the same table. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2405) [hbase] Merge region tool exposed in shell and/or in UI
[ https://issues.apache.org/jira/browse/HADOOP-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556694#action_12556694 ] Bryan Duxbury commented on HADOOP-2405: --- So, you envision the merge operation to not only require manual triggering but to require manual targeting? Shouldn't the point of merging regions be to maintain the equilibrium of size of regions? Under what circumstances will you have to manually intervene to keep regions appropriately sized? It seems like this should really only happen after a substantial number of deletions has occurred, right? That would cause a compacted region to shrink below a healthy size, and if it could be merged with a neighbor, it would be nice. This logic should be built in and automatic, otherwise it would require constant monitoring of region sizes by an administrator. Other than this sort of automatic merging, when would you want to manually merge two regions? Doesn't that expose a somewhat dangerous amount of functionality to the end user? > [hbase] Merge region tool exposed in shell and/or in UI > --- > > Key: HADOOP-2405 > URL: https://issues.apache.org/jira/browse/HADOOP-2405 > Project: Hadoop > Issue Type: New Feature > Components: contrib/hbase >Reporter: stack >Priority: Minor > > hbase has support for merging regions. Expose a merge trigger in the shell > or in the UI (Can only merge adjacent features so perhaps only makes sense in > UI in the regionserver UI). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2364) when hbase regionserver restarts, it says "impossible state for createLease()"
[ https://issues.apache.org/jira/browse/HADOOP-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556655#action_12556655 ] Bryan Duxbury commented on HADOOP-2364: --- Did this actually cause any problems? Did the Region Server continue to start up ok, or is this just a spurious error message? > when hbase regionserver restarts, it says "impossible state for createLease()" > -- > > Key: HADOOP-2364 > URL: https://issues.apache.org/jira/browse/HADOOP-2364 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase >Affects Versions: 0.16.0 >Reporter: Michael Bieniosek >Priority: Minor > > I restarted a regionserver, and got this error in its logs: > org.apache.hadoop.ipc.RemoteException: java.io.IOException: > java.lang.AssertionError: Impossible state for createLease(): Lease > -435227488/-435227488 is still held. > at org.apache.hadoop.hbase.Leases.createLease(Leases.java:145) > at > org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1278 > ) > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596) > at org.apache.hadoop.ipc.Client.call(Client.java:482) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184) > at $Proxy0.regionServerStartup(Unknown Source) > at > org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.jav > a:1025) > at org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:659) > at java.lang.Thread.run(Unknown Source) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
[ https://issues.apache.org/jira/browse/HADOOP-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556020#action_12556020 ] Bryan Duxbury commented on HADOOP-2518: --- After some discussion with Jim, it looks like this patch is going to be put on hold. After he's checked in his patch to change file organization, we can re-address this desire. > [hbase] Refactor HStore and internal (and related) classes into hbase.store > subpackage > -- > > Key: HADOOP-2518 > URL: https://issues.apache.org/jira/browse/HADOOP-2518 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2518.patch > > > HStore, the class responsible for managing data storage for column families > within a region, is a very large class with more than a few internal classes. > I propose moving HStore and HStoreFile into the hbase.store subpackage, and > then moving internal classes out of HStore and into their own > package-internal classes within the subpackage. This will make the code much > more readable and manageable. In addition, it will help to logically > delineate which classes are involved in managing a store. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2521) [hbase] HStoreFiles needlessly store the column family name in every entry
[ https://issues.apache.org/jira/browse/HADOOP-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555997#action_12555997 ] Bryan Duxbury commented on HADOOP-2521: --- Jim, I am aware that multiple qualified cells show up in the same store file per row in the same HStoreFile. I'm just suggesting that the part that comes before the qualified name is unnecessary. I understand that changing this would necessitate adding a new key type and transform logic, but I'm not convinced that the translation would actually cost that much more time. You have to recognize that even though the data is precomputed, it is probably coming off of a disk on another computer through the network in 64MB blocks. I have to think that the added transmission time of all the redundant data in aggregate is at least as much as the added time it would take to do translation, and possibly more. > [hbase] HStoreFiles needlessly store the column family name in every entry > -- > > Key: HADOOP-2521 > URL: https://issues.apache.org/jira/browse/HADOOP-2521 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Priority: Minor > > Today, HStoreFiles keep the entire serialized HStoreKey objects around for > every cell in the HStore. Since HStores are 1-1 with column families, this is > really unnecessary - you can always surmise the column family by looking at > the HStore it belongs to. (This information would ostensibly come from the > file name or a header section.) This means that we could remove the column > family part of the HStoreKeys we put into the HStoreFile, reducing the size > of data stored. This would be a space-saving benefit, removing redundant > data, and could be a speed benefit, as you have to scan over less data in > memory and transfer less data over the network. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2507) [hbase] REST servlet does not properly base64 row keys and column names
[ https://issues.apache.org/jira/browse/HADOOP-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2507: -- Status: Open (was: Patch Available) Doesn't look like hudson is paying attention to me. > [hbase] REST servlet does not properly base64 row keys and column names > --- > > Key: HADOOP-2507 > URL: https://issues.apache.org/jira/browse/HADOOP-2507 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury >Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2507.patch > > > At the moment, the REST servlet treats row keys and column names as though > they are all printable, when in fact they can definitely be arbitrary binary > strings. When returning these items, they should be base64 encoded like cell > values are. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2507) [hbase] REST servlet does not properly base64 row keys and column names
[ https://issues.apache.org/jira/browse/HADOOP-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2507: -- Status: Patch Available (was: Open) Trying again. > [hbase] REST servlet does not properly base64 row keys and column names > --- > > Key: HADOOP-2507 > URL: https://issues.apache.org/jira/browse/HADOOP-2507 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2507.patch > > > At the moment, the REST servlet treats row keys and column names as though > they are all printable, when in fact they can definitely be arbitrary binary > strings. When returning these items, they should be base64 encoded like cell > values are. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2521) [hbase] HStoreFiles needlessly store the column family name in every entry
[hbase] HStoreFiles needlessly store the column family name in every entry -- Key: HADOOP-2521 URL: https://issues.apache.org/jira/browse/HADOOP-2521 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Priority: Minor Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (HADOOP-2519) [hbase] Customized RPC
So, you created a brand-new RPC framework? Is there a reason you took this approach over trying out something like Thrift? Thrift is binary and probably lighter weight than the original Hadoop IPC. It also has the advantage of (maybe) being easier to add on to as the RPC interface grows. On Jan 3, 2008, at 1:58 PM, stack (JIRA) wrote: [hbase] Customized RPC -- Key: HADOOP-2519 URL: https://issues.apache.org/jira/browse/ HADOOP-2519 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: stack Priority: Minor Fix For: 0.16.0 A custom RPC that passes codes representing class names instead of class-as-String saves passing plus encoding/decoding classname Strings; it makes the PerformanceEvaluation sequentialWrites test almost 3 times faster and random reads > 30% faster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
[ https://issues.apache.org/jira/browse/HADOOP-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2518: -- Attachment: 2518.patch Here's my go at the refactor described above. Files modified: D src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHStoreFile.java D src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHMemcache.java A src/contrib/hbase/src/test/org/apache/hadoop/hbase/store A src/contrib/hbase/src/test/org/apache/hadoop/hbase/store/TestHStoreFile.java A src/contrib/hbase/src/test/org/apache/hadoop/hbase/store/TestHMemcache.java M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestCompaction.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogKey.java D src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java D src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/BloomFilterDescriptor.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLog.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/CompactionReader.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStoreSize.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/Memcache.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStore.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStoreFile.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/StoreFileScanner.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/MapFileCompactionReader.java A src/contrib/hbase/src/java/org/apache/hadoop/hbase/store/HStoreScanner.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HAbstractScanner.java M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java > [hbase] Refactor HStore and internal (and related) classes into hbase.store > subpackage > -- > > Key: HADOOP-2518 > URL: https://issues.apache.org/jira/browse/HADOOP-2518 > Project: Hadoop > Issue Type: Improvement > Components: contrib/hbase >Reporter: Bryan Duxbury >Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2518.patch > > > HStore, the class responsible for managing data storage for column families > within a region, is a very large class with more than a few internal classes. > I propose moving HStore and HStoreFile into the hbase.store subpackage, and > then moving internal classes out of HStore and into their own > package-internal classes within the subpackage. This will make the code much > more readable and manageable. In addition, it will help to logically > delineate which classes are involved in managing a store. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2518) [hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage
[hbase] Refactor HStore and internal (and related) classes into hbase.store subpackage -- Key: HADOOP-2518 URL: https://issues.apache.org/jira/browse/HADOOP-2518 Project: Hadoop Issue Type: Improvement Components: contrib/hbase Reporter: Bryan Duxbury Assignee: Bryan Duxbury Priority: Minor HStore, the class responsible for managing data storage for column families within a region, is a very large class with more than a few internal classes. I propose moving HStore and HStoreFile into the hbase.store subpackage, and then moving internal classes out of HStore and into their own package-internal classes within the subpackage. This will make the code much more readable and manageable. In addition, it will help to logically delineate which classes are involved in managing a store. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-2513) [hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile
[hbase] HStore#get and HStore#getFull may not return expected values by timestamp when there is more than one MapFile - Key: HADOOP-2513 URL: https://issues.apache.org/jira/browse/HADOOP-2513 Project: Hadoop Issue Type: Bug Components: contrib/hbase Reporter: Bryan Duxbury Ok, this one is a little tricky. Let's say that you write a row with some value without a timestamp, thus meaning right now. Then, the memcache gets flushed out to a MapFile. Then, you write another value to the same row, this time with a timestamp that is in the past, ie, before the "now" timestamp of the first put. Some time later, but before there is a compaction, if you do a get for this row, and only ask for a single version, you will logically be expecting the latest version of the cell, which you would assume would be the one written at "now" time. Instead, you will get the value written into the "past" cell, because even though it is tagged as having happened in the past, it actually *was written* after the "now" cell, and thus when #get searches for satisfying values, it runs into the one most recently written first. The result of this problem is inconsistent data results. Note that this problem only ever exists when there's an uncompacted HStore, because during compaction, these cells will all get sorted into the correct order by timestamp and such. In a way, this actually makes the problem worse, because then you could easily get inconsistent results from HBase about the same (unchanged) row depending on whether there's been a flush/compaction. The only solution I can think of for this problem at the moment is to scan all the MapFiles and Memcache for possible results, sort them, and then select the desired number of versions off of the top. This is unfortunate because it means you never get the snazzy shortcircuit logic except within a single mapfile or memcache. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2504) [hbase] REST servlet method for deleting a scanner was not properly mapped
[ https://issues.apache.org/jira/browse/HADOOP-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2504: -- Status: Patch Available (was: Open) Resending to Hudson, as it didn't seem to get built. > [hbase] REST servlet method for deleting a scanner was not properly mapped > -- > > Key: HADOOP-2504 > URL: https://issues.apache.org/jira/browse/HADOOP-2504 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury >Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2504-2.patch, 2504.patch > > > As reported by Billy Pearson, the code for DELETE on an existing scanner was > written, but not properly mapped to real methods after the REST refactoring > work was done. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2504) [hbase] REST servlet method for deleting a scanner was not properly mapped
[ https://issues.apache.org/jira/browse/HADOOP-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Duxbury updated HADOOP-2504: -- Status: Open (was: Patch Available) > [hbase] REST servlet method for deleting a scanner was not properly mapped > -- > > Key: HADOOP-2504 > URL: https://issues.apache.org/jira/browse/HADOOP-2504 > Project: Hadoop > Issue Type: Bug > Components: contrib/hbase > Reporter: Bryan Duxbury > Assignee: Bryan Duxbury >Priority: Minor > Attachments: 2504-2.patch, 2504.patch > > > As reported by Billy Pearson, the code for DELETE on an existing scanner was > written, but not properly mapped to real methods after the REST refactoring > work was done. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
