[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding
[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485834#comment-13485834
]
Hadoop QA commented on HBASE-4676:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551154/HBASE-4676-prefix-tree-trunk-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 143
new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
102 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 49 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3170//console
This message is automatically generated.
> Prefix Compression - Trie data block encoding
> -
>
> Key: HBASE-4676
> URL: https://issues.apache.org/jira/browse/HBASE-4676
> Project: HBase
> Issue Type: New Feature
> Components: io, Performance, regionserver
>Affects Versions: 0.96.0
>Reporter: Matt Corgan
>Assignee: Matt Corgan
> Attachments: HBASE-4676-0.94-v1.patch,
> HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch,
> HBASE-4676-prefix-tree-trunk-v3.patch, hbase-prefix-trie-0.1.jar,
> PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
> blockSize.png
>
>
> The HBase data block format has room for 2 significant improvements for
> applications that have high block cache hit ratios.
> First, there is no prefix compression, and the current KeyValue format is
> somewhat metadata heavy, so there can be tremendous memory bloat for many
> common data layouts, specifically those with long keys and short values.
> Second, there is no random access to KeyValues inside data blocks. This
> means that every time you double the datablock size, average seek time (or
> average cpu consumption) goes up by a factor of 2. The standard 64KB block
> size is ~10x slower for random seeks than a 4KB block size, but block sizes
> as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB
> or more may be more efficient from a disk access and block-cache perspective
> in many big-data applications, but doing so is infeasible from a random seek
> perspective.
> The PrefixTrie block encoding format attempts to solve both of these
> problems. Some features:
> * trie format for row key encoding completely eliminates duplicate row keys
> and encodes similar row keys into a standard trie structure which also saves
> a lot of space
> * the column family is currently stored once at the beginning of each block.
> this could easily be modified to allow multiple family names per block
> * all qualifiers in the block are stored in their own trie format which
> caters nicely to wide rows. duplicate qualifers between rows are eliminated.
> the size of this trie determines the width of the block's qualifier
> fixed-width-int
> * the minimum timestamp is stored at the beginning of the block, and deltas
> are calculated from that. the maximum delta determines the width of the
> block's timestamp fixed-width-int
> The block is structured with metadata at the beginning, then a section for
>
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[
https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485830#comment-13485830
]
ramkrishna.s.vasudevan commented on HBASE-6060:
---
Good review Rajesh.
Yes the OPEN thing seems to be handled. Also we have the code guarding
{code}
else if (addressFromAM != null
&& !addressFromAM.equals(this.serverName)) {
LOG.debug("Skip assigning region "
+ e.getKey().getRegionNameAsString()
+ " because it has been opened in "
+ addressFromAM.getServerName());
{code}
if the server name got changed just after OPEN got processed. This should be
fine.
> Regions's in OPENING state from failed regionservers takes a long time to
> recover
> -
>
> Key: HBASE-6060
> URL: https://issues.apache.org/jira/browse/HBASE-6060
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Enis Soztutar
>Assignee: rajeshbabu
> Fix For: 0.92.3, 0.94.3, 0.96.0
>
> Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch,
> 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt,
> 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch,
> 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch,
> 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch,
> HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch,
> HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch,
> trunk-6060.patch, trunk-6060_v2.patch
>
>
> we have seen a pattern in tests, that the regions are stuck in OPENING state
> for a very long time when the region server who is opening the region fails.
> My understanding of the process:
>
> - master calls rs to open the region. If rs is offline, a new plan is
> generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in
> master memory, zk still shows OFFLINE). See HRegionServer.openRegion(),
> HMaster.assign()
> - RegionServer, starts opening a region, changes the state in znode. But
> that znode is not ephemeral. (see ZkAssign)
> - Rs transitions zk node from OFFLINE to OPENING. See
> OpenRegionHandler.process()
> - rs then opens the region, and changes znode from OPENING to OPENED
> - when rs is killed between OPENING and OPENED states, then zk shows OPENING
> state, and the master just waits for rs to change the region state, but since
> rs is down, that wont happen.
> - There is a AssignmentManager.TimeoutMonitor, which does exactly guard
> against these kind of conditions. It periodically checks (every 10 sec by
> default) the regions in transition to see whether they timedout
> (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min,
> which explains what you and I are seeing.
> - ServerShutdownHandler in Master does not reassign regions in OPENING
> state, although it handles other states.
> Lowering that threshold from the configuration is one option, but still I
> think we can do better.
> Will investigate more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[
https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485827#comment-13485827
]
Hadoop QA commented on HBASE-3909:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551148/3909-102812.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
89 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3169//console
This message is automatically generated.
> Add dynamic config
> --
>
> Key: HBASE-3909
> URL: https://issues.apache.org/jira/browse/HBASE-3909
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Fix For: 0.96.0
>
> Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909.v1,
> 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch
>
>
> I'm sure this issue exists already, at least as part of the discussion around
> making online schema edits possible, but no hard this having its own issue.
> Ted started a conversation on this topic up on dev and Todd suggested we
> lookd at how Hadoop did it over in HADOOP-7001
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Corgan updated HBASE-4676: --- Attachment: HBASE-4676-prefix-tree-trunk-v3.patch HBASE-4676-prefix-tree-trunk-v3.patch fixes a memstoreTS related bug. Submitting this patch to see if TestHFileDataBlockEncoder still fails in jenkins (since it passes locally for me). > Prefix Compression - Trie data block encoding > - > > Key: HBASE-4676 > URL: https://issues.apache.org/jira/browse/HBASE-4676 > Project: HBase > Issue Type: New Feature > Components: io, Performance, regionserver >Affects Versions: 0.96.0 >Reporter: Matt Corgan >Assignee: Matt Corgan > Attachments: HBASE-4676-0.94-v1.patch, > HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, > HBASE-4676-prefix-tree-trunk-v3.patch, hbase-prefix-trie-0.1.jar, > PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by > blockSize.png > > > The HBase data block format has room for 2 significant improvements for > applications that have high block cache hit ratios. > First, there is no prefix compression, and the current KeyValue format is > somewhat metadata heavy, so there can be tremendous memory bloat for many > common data layouts, specifically those with long keys and short values. > Second, there is no random access to KeyValues inside data blocks. This > means that every time you double the datablock size, average seek time (or > average cpu consumption) goes up by a factor of 2. The standard 64KB block > size is ~10x slower for random seeks than a 4KB block size, but block sizes > as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB > or more may be more efficient from a disk access and block-cache perspective > in many big-data applications, but doing so is infeasible from a random seek > perspective. > The PrefixTrie block encoding format attempts to solve both of these > problems. Some features: > * trie format for row key encoding completely eliminates duplicate row keys > and encodes similar row keys into a standard trie structure which also saves > a lot of space > * the column family is currently stored once at the beginning of each block. > this could easily be modified to allow multiple family names per block > * all qualifiers in the block are stored in their own trie format which > caters nicely to wide rows. duplicate qualifers between rows are eliminated. > the size of this trie determines the width of the block's qualifier > fixed-width-int > * the minimum timestamp is stored at the beginning of the block, and deltas > are calculated from that. the maximum delta determines the width of the > block's timestamp fixed-width-int > The block is structured with metadata at the beginning, then a section for > the row trie, then the column trie, then the timestamp deltas, and then then > all the values. Most work is done in the row trie, where every leaf node > (corresponding to a row) contains a list of offsets/references corresponding > to the cells in that row. Each cell is fixed-width to enable binary > searching and is represented by [1 byte operationType, X bytes qualifier > offset, X bytes timestamp delta offset]. > If all operation types are the same for a block, there will be zero per-cell > overhead. Same for timestamps. Same for qualifiers when i get a chance. > So, the compression aspect is very strong, but makes a few small sacrifices > on VarInt size to enable faster binary searches in trie fan-out nodes. > A more compressed but slower version might build on this by also applying > further (suffix, etc) compression on the trie nodes at the cost of slower > write speed. Even further compression could be obtained by using all VInts > instead of FInts with a sacrifice on random seek speed (though not huge). > One current drawback is the current write speed. While programmed with good > constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not > programmed with the same level of optimization as the read path. Work will > need to be done to optimize the data structures used for encoding and could > probably show a 10x increase. It will still be slower than delta encoding, > but with a much higher decode speed. I have not yet created a thorough > benchmark for write speed nor sequential read speed. > Though the trie is reaching a point where it is internally very efficient > (probably within half or a quarter of its max read speed) the way that hbase > currently uses it is far from optimal. The KeyValueScanner and related > classes that iterate through the trie will eventually need to be smarter and > have methods to do things like skipping to the next row of results without >
[jira] [Commented] (HBASE-3909) Add dynamic config
[
https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485812#comment-13485812
]
Ted Yu commented on HBASE-3909:
---
The test passed locally with patch on my MacBook:
{code}
Running org.apache.hadoop.hbase.TestMultiVersions
2012-10-28 20:41:05.379 java[36641:1903] Unable to load realm mapping info from
SCDynamicStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.007 sec
{code}
> Add dynamic config
> --
>
> Key: HBASE-3909
> URL: https://issues.apache.org/jira/browse/HBASE-3909
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Fix For: 0.96.0
>
> Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909.v1,
> 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch
>
>
> I'm sure this issue exists already, at least as part of the discussion around
> making online schema edits possible, but no hard this having its own issue.
> Ted started a conversation on this topic up on dev and Todd suggested we
> lookd at how Hadoop did it over in HADOOP-7001
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-3909: Attachment: 3909-102812.patch > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: Bug >Reporter: stack > Fix For: 0.96.0 > > Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909.v1, > 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485805#comment-13485805 ] Subbu M Iyer commented on HBASE-3909: - Attaching latest patch that addresses the review comments. All tests pass except the following and I am reviewing the failures. Tests in error: testGetRowVersions(org.apache.hadoop.hbase.TestMultiVersions): Shutting down testScanMultipleVersions(org.apache.hadoop.hbase.TestMultiVersions): org.apache.hadoop.hbase.MasterNotRunningException: Can create a proxy to master, but it is not running > Add dynamic config > -- > > Key: HBASE-3909 > URL: https://issues.apache.org/jira/browse/HBASE-3909 > Project: HBase > Issue Type: Bug >Reporter: stack > Fix For: 0.96.0 > > Attachments: 3909_090712-2.patch, 3909-102812.patch, 3909.v1, > 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch > > > I'm sure this issue exists already, at least as part of the discussion around > making online schema edits possible, but no hard this having its own issue. > Ted started a conversation on this topic up on dev and Todd suggested we > lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6222) Add per-KeyValue Security
[
https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485788#comment-13485788
]
Jerry Chen commented on HBASE-6222:
---
[~saint@gmail.com]
bq. A put that broadened visibility would be for the current put only? How
would it effect already-put values?
When the visibility is part of the column key, a broadened visibility will not
affect the existing columns that with the same {rowid, family, qualifier} but
with different visibilities. Thus, the put will only affect the columns that
has the same {rowid, family, qualifier, visibility}. Different visibilities has
the same effect as different qualifiers.
While as to DeleteFamily or DeleteColumn, Accumulo doesn't have such as
operations. It has only Delete mutation to delete a specified {rowid, family,
qualifier, visibility}. The idea to keep DeleteFamily and DeleteColumn still
working with visibility in HBase is that A DeleteFamily operation now will only
affect "all columns in this family with the specified visibility" other than
originally "all columns in this family". The same with DeleteColumn.
One thing to consider if the visibility is part of the key. As there are
suggestions to provide support for general tags for KV so that not only
visibility tags can be stored in it, but also other tags that needed in the
future can be added easily. Will general tags concept (comparing to a
visibility tag) makes the concept of the column key too complex?
> Add per-KeyValue Security
> -
>
> Key: HBASE-6222
> URL: https://issues.apache.org/jira/browse/HBASE-6222
> Project: HBase
> Issue Type: New Feature
> Components: security
>Reporter: stack
>
> Saw an interesting article:
> http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14
> "The Senate Armed Services Committee version of the fiscal 2013 national
> defense authorization act (S. 3254) would require DoD agencies to foreswear
> the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO
> certifies that there exists either no viable commercial open source database
> with security features comparable to [Accumulo] (such as the HBase or
> Cassandra databases)..."
> Not sure what a 'commercial open source database' is, and I'm not sure whats
> going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security'
> like Accumulo's, we might put ourselves in the running for federal
> contributions?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485760#comment-13485760 ] Jesse Yates commented on HBASE-6707: Well, at least it didn't fail immediately. (cross-fingers and knocks on wood). > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6707-v4-addendum.txt, hbase-6707-v0.patch, > hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch, > hbase-6707-v4-addendum.patch, hbase-6707-v4.patch, > testZooKeeperTableArchiveClient-output.txt > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6665) ROOT region should not be splitted even with META row as explicit split key
[ https://issues.apache.org/jira/browse/HBASE-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485758#comment-13485758 ] Hudson commented on HBASE-6665: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #243 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/243/]) HBASE-6665 ROOT region should not be splitted even with META row as explicit split key (Rajesh) (Revision 1403037) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > ROOT region should not be splitted even with META row as explicit split key > --- > > Key: HBASE-6665 > URL: https://issues.apache.org/jira/browse/HBASE-6665 > Project: HBase > Issue Type: Bug >Reporter: Y. SREENIVASULU REDDY >Assignee: rajeshbabu > Fix For: 0.96.0 > > Attachments: HBASE-6665_trunk.patch > > > split operation on ROOT table by specifying explicit split key as .META. > closing the ROOT region and taking long time to fail the split before > rollback. > I think we can skip split for ROOT table as how we are doing for META region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485757#comment-13485757 ] Hudson commented on HBASE-6707: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #243 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/243/]) HBASE-6707 TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps; ADDENDUM v4 (Revision 1403098) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6707-v4-addendum.txt, hbase-6707-v0.patch, > hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch, > hbase-6707-v4-addendum.patch, hbase-6707-v4.patch, > testZooKeeperTableArchiveClient-output.txt > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[
https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485756#comment-13485756
]
Ted Yu commented on HBASE-6597:
---
Update on my recent fidings.
I came up with patch for 0.94 branch.
Most data block encoding related tests pass.
TestHFileBlockCompatibility poses a little challenge. There is no embedded
checksum feature in 0.89-fb branch. So this test is unique to 0.94 / trunk.
In the test, there is a copy of Writer class which I assume shouldn't be
modified, at least not for a point release.
The test reuses some code from TestHFileBlock.java where there is some change
related to usage of Writer:
{code}
- static int writeTestKeyValues(OutputStream dos, int seed, boolean
includesMemstoreTS)
+ static void writeTestKeyValues(OutputStream dos, Writer hbw, int seed,
boolean includesMemstoreTS)
{code}
This is the test failure I am observing now:
{code}
testDataBlockEncoding[0](org.apache.hadoop.hbase.io.hfile.TestHFileBlockCompatibility)
Time elapsed: 0.129 sec <<< FAILURE!
org.junit.ComparisonFailure: Content mismath for compression NONE, encoding
PREFIX, pread false, commonPrefix 2, expected length 1859, actual length 1859
expected:<\x00\x00\x0[B\xB8]*\x0A\x00\x00\x0A\x0...> but
was:<\x00\x00\x0[0\x00]*\x0A\x00\x00\x0A\x0...>
at org.junit.Assert.assertEquals(Assert.java:125)
at
org.apache.hadoop.hbase.io.hfile.TestHFileBlock.assertBuffersEqual(TestHFileBlock.java:463)
at
org.apache.hadoop.hbase.io.hfile.TestHFileBlockCompatibility.testDataBlockEncoding(TestHFileBlockCompatibility.java:337)
{code}
> Block Encoding Size Estimation
> --
>
> Key: HBASE-6597
> URL: https://issues.apache.org/jira/browse/HBASE-6597
> Project: HBase
> Issue Type: Improvement
> Components: io
>Affects Versions: 0.89-fb
>Reporter: Brian Nixon
>Assignee: Mikhail Bautin
>Priority: Minor
> Attachments: 6597-trunk.txt, D5895.1.patch, D5895.2.patch,
> D5895.3.patch, D5895.4.patch, D5895.5.patch
>
>
> Blocks boundaries as created by current writers are determined by the size of
> the unencoded data. However, blocks in memory are kept encoded. By using an
> estimate for the encoded size of the block, we can get greater consistency in
> size.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485739#comment-13485739 ] Hudson commented on HBASE-6707: --- Integrated in HBase-TRUNK #3494 (See [https://builds.apache.org/job/HBase-TRUNK/3494/]) HBASE-6707 TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps; ADDENDUM v4 (Revision 1403098) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6707-v4-addendum.txt, hbase-6707-v0.patch, > hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch, > hbase-6707-v4-addendum.patch, hbase-6707-v4.patch, > testZooKeeperTableArchiveClient-output.txt > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485724#comment-13485724 ] stack commented on HBASE-6707: -- Committed v4. Lets see how it does. > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6707-v4-addendum.txt, hbase-6707-v0.patch, > hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch, > hbase-6707-v4-addendum.patch, hbase-6707-v4.patch, > testZooKeeperTableArchiveClient-output.txt > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485721#comment-13485721
]
Hadoop QA commented on HBASE-4583:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12551123/4583-trunk-v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3168//console
This message is automatically generated.
> Integrate RWCC with Append and Increment operations
> ---
>
> Key: HBASE-4583
> URL: https://issues.apache.org/jira/browse/HBASE-4583
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 4583-trunk-radical.txt, 4583-trunk-radical_v2.txt,
> 4583-trunk-v3.txt, 4583.txt, 4583-v2.txt, 4583-v3.txt, 4583-v4.txt
>
>
> Currently Increment and Append operations do not work with RWCC and hence a
> client could see the results of multiple such operation mixed in the same
> Get/Scan.
> The semantics might be a bit more interesting here as upsert adds and removes
> to and from the memstore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Attachment: 4583-trunk-v3.txt This makes it even more like all other other operation: Use the existing applyFamilyMapToMemstore method to make the actual changes to the memstore. > Integrate RWCC with Append and Increment operations > --- > > Key: HBASE-4583 > URL: https://issues.apache.org/jira/browse/HBASE-4583 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.3, 0.96.0 > > Attachments: 4583-trunk-radical.txt, 4583-trunk-radical_v2.txt, > 4583-trunk-v3.txt, 4583.txt, 4583-v2.txt, 4583-v3.txt, 4583-v4.txt > > > Currently Increment and Append operations do not work with RWCC and hence a > client could see the results of multiple such operation mixed in the same > Get/Scan. > The semantics might be a bit more interesting here as upsert adds and removes > to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485704#comment-13485704
]
Lars Hofhansl commented on HBASE-7051:
--
BTW, the approach I took in HBASE-4583 is akin to #1.
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
> Fix For: 0.94.3, 0.96.0
>
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6796) Backport HBASE-5547, Don't delete HFiles in backup mode.
[ https://issues.apache.org/jira/browse/HBASE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485695#comment-13485695 ] Jesse Yates commented on HBASE-6796: [~lhofhansl] what do you think, wanna pull this into 0.94.3? > Backport HBASE-5547, Don't delete HFiles in backup mode. > > > Key: HBASE-6796 > URL: https://issues.apache.org/jira/browse/HBASE-6796 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Fix For: 0.94.3 > > Attachments: hbase-5547-0.94-backport-v0.patch, hbase-6796-v0.patch > > > See HBASE-5547 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485687#comment-13485687 ] Jesse Yates commented on HBASE-6707: [~stack] i'd say commit v4-addendum.patch. > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6707-v4-addendum.txt, hbase-6707-v0.patch, > hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch, > hbase-6707-v4-addendum.patch, hbase-6707-v4.patch, > testZooKeeperTableArchiveClient-output.txt > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6665) ROOT region should not be splitted even with META row as explicit split key
[ https://issues.apache.org/jira/browse/HBASE-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485681#comment-13485681 ] Hudson commented on HBASE-6665: --- Integrated in HBase-TRUNK #3493 (See [https://builds.apache.org/job/HBase-TRUNK/3493/]) HBASE-6665 ROOT region should not be splitted even with META row as explicit split key (Rajesh) (Revision 1403037) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > ROOT region should not be splitted even with META row as explicit split key > --- > > Key: HBASE-6665 > URL: https://issues.apache.org/jira/browse/HBASE-6665 > Project: HBase > Issue Type: Bug >Reporter: Y. SREENIVASULU REDDY >Assignee: rajeshbabu > Fix For: 0.96.0 > > Attachments: HBASE-6665_trunk.patch > > > split operation on ROOT table by specifying explicit split key as .META. > closing the ROOT region and taking long time to fail the split before > rollback. > I think we can skip split for ROOT table as how we are doing for META region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6665) ROOT region should not be splitted even with META row as explicit split key
[ https://issues.apache.org/jira/browse/HBASE-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6665: -- Fix Version/s: 0.96.0 Assignee: rajeshbabu Hadoop Flags: Reviewed Integrated to trunk. Thanks for the patch, Rajesh. Thanks for the review, Stack. > ROOT region should not be splitted even with META row as explicit split key > --- > > Key: HBASE-6665 > URL: https://issues.apache.org/jira/browse/HBASE-6665 > Project: HBase > Issue Type: Bug >Reporter: Y. SREENIVASULU REDDY >Assignee: rajeshbabu > Fix For: 0.96.0 > > Attachments: HBASE-6665_trunk.patch > > > split operation on ROOT table by specifying explicit split key as .META. > closing the ROOT region and taking long time to fail the split before > rollback. > I think we can skip split for ROOT table as how we are doing for META region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485635#comment-13485635 ] Jarek Jarcec Cecho commented on HBASE-6929: --- As of now, Sqoop is simply skipping HBase tests on Hadoop 2, so we do have working workaround and we could potentially wait. But I would personally prefer to solve it as soon as possible as I do not like current situation :-/ Having also HBase 0.92 test jars compiled against Hadoop 2 would be nice, so that we could test against them as well and make sure that everything is working as expected. Jarcec > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3925) Make Shell's -d and debug cmd behave the same
[
https://issues.apache.org/jira/browse/HBASE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485610#comment-13485610
]
Hudson commented on HBASE-3925:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #242 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/242/])
HBASE-3925 Make Shell's -d and debug cmd behave the same (Revision 1402923)
Result = FAILURE
stack :
Files :
* /hbase/trunk/bin/hirb.rb
> Make Shell's -d and debug cmd behave the same
> -
>
> Key: HBASE-3925
> URL: https://issues.apache.org/jira/browse/HBASE-3925
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 0.90.3, 0.90.7, 0.92.2, 0.94.3, 0.96.0, 0.98.0
>Reporter: Lars George
>Assignee: liang xie
>Priority: Trivial
> Labels: patch
> Fix For: 0.96.0
>
> Attachments: HBASE-3925.patch, HBASE-3925-v2.txt
>
>
> The -d option switches log4j to DEBUG and leaves the backtrace level at the
> default. When using the supplied debug command we only switch the backtrace,
> but I would think this also should set the log4j levels:
> {noformat}
> # Debugging method
> def debug
> if @shell.debug
> @shell.debug = false
> conf.back_trace_limit = 0
> else
> @shell.debug = true
> conf.back_trace_limit = 100
> end
> debug?
> end
> {noformat}
> could be something like
> {noformat}
> # Debugging method
> def debug
> if @shell.debug
> @shell.debug = false
> conf.back_trace_limit = 0
> log_level = org.apache.log4j.Level::ERROR
> else
> @shell.debug = true
> conf.back_trace_limit = 100
> log_level = org.apache.log4j.Level::DEBUG
> end
>
> org.apache.log4j.Logger.getLogger("org.apache.zookeeper").setLevel(log_level)
>
> org.apache.log4j.Logger.getLogger("org.apache.hadoop.hbase").setLevel(log_level)
> debug?
> end
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[
https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485607#comment-13485607
]
rajeshbabu commented on HBASE-6060:
---
@Jimmy,
Patch looks good and it properly reassigns the regions transitioning(which are
in PENDING_OPEN,OPENING state) to dead server by TM and also avoids double
assignments.
But the regions which are in OPEN state may not be assigned.
{code}
RegionState rit = services.getAssignmentManager()
.getRegionStates().getRegionTransitionState(e.getKey());
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
ServerName addressFromAM = this.services.getAssignmentManager()
.getRegionStates().getRegionServerOfRegion(e.getKey());
if (rit != null && !rit.isClosing() && !rit.isPendingClose() &&
!rit.isSplitting()) {
// Skip regions that were in transition unless CLOSING or
// PENDING_CLOSE
LOG.info("Skip assigning region " + rit.toString());
{code}
If the region state is OPEN and server in RIT same as the dead server we can
add it to toAssignRegions to assign.
-> Before forcing region state to offline we are checking current state and if
its other than CLOSED or OFFLINE(forceNewPlan is true in case of assign call
from TM) we calling unassing on the region.
if the server in the RIT is dead we are deleting znode if its in
M_ZK_REGION_CLOSING or RS_ZK_REGION_CLOSED but in case of
PENDING_OPEN,OPENING,OPEN case znode state wont be both. May be we can avoid
two network calls to zookeeper here by having region state check.
{code}
if (!serverManager.isServerOnline(server)) {
// delete the node. if no node exists need not bother.
deleteClosingOrClosedNode(region);
regionOffline(region);
return;
}
{code}
Please correct me if wrong. Great work Jimmy.
> Regions's in OPENING state from failed regionservers takes a long time to
> recover
> -
>
> Key: HBASE-6060
> URL: https://issues.apache.org/jira/browse/HBASE-6060
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Enis Soztutar
>Assignee: rajeshbabu
> Fix For: 0.92.3, 0.94.3, 0.96.0
>
> Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch,
> 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt,
> 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch,
> 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch,
> 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch,
> HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch,
> HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch,
> trunk-6060.patch, trunk-6060_v2.patch
>
>
> we have seen a pattern in tests, that the regions are stuck in OPENING state
> for a very long time when the region server who is opening the region fails.
> My understanding of the process:
>
> - master calls rs to open the region. If rs is offline, a new plan is
> generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in
> master memory, zk still shows OFFLINE). See HRegionServer.openRegion(),
> HMaster.assign()
> - RegionServer, starts opening a region, changes the state in znode. But
> that znode is not ephemeral. (see ZkAssign)
> - Rs transitions zk node from OFFLINE to OPENING. See
> OpenRegionHandler.process()
> - rs then opens the region, and changes znode from OPENING to OPENED
> - when rs is killed between OPENING and OPENED states, then zk shows OPENING
> state, and the master just waits for rs to change the region state, but since
> rs is down, that wont happen.
> - There is a AssignmentManager.TimeoutMonitor, which does exactly guard
> against these kind of conditions. It periodically checks (every 10 sec by
> default) the regions in transition to see whether they timedout
> (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min,
> which explains what you and I are seeing.
> - ServerShutdownHandler in Master does not reassign regions in OPENING
> state, although it handles other states.
> Lowering that threshold from the configuration is one option, but still I
> think we can do better.
> Will investigate more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
