[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700870#comment-14700870 ] Jingcheng Du commented on HBASE-14227: -- Agree with [~anoopsamjohn]. compactMob only compacts the mob files. The store files are NOT included in this operations. We can allow less MOB compaction and even disable it. Compacting the normal CF and MOB CF is not a good idea because of mob's big size. The mob APIs can be removed from Admin, and are folded into existing ones. How about using a special name format, for example columnFamily$MOB? So we can know it is used to trigger a mob compaction on a mob-enabled column family? Thanks. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14234) Exception encountered in WALProcedureStore#rollWriter() should be properly handled
[
https://issues.apache.org/jira/browse/HBASE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700884#comment-14700884
]
Hadoop QA commented on HBASE-14234:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750944/14234-v1.txt
against master branch at commit 395ec5a9bb48324a8b7dd61790a954a2998a8f80.
ATTACHMENT ID: 12750944
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile.testLosingFileDuringScan(TestCorruptedRegionStoreFile.java:147)
at
org.apache.hadoop.hbase.regionserver.TestRegionReplicas.testRefreshStoreFiles(TestRegionReplicas.java:237)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testLingeringSplitParent(TestHBaseFsck.java:1695)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15136//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15136//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15136//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15136//console
This message is automatically generated.
> Exception encountered in WALProcedureStore#rollWriter() should be properly
> handled
> --
>
> Key: HBASE-14234
> URL: https://issues.apache.org/jira/browse/HBASE-14234
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 14234-v1.txt
>
>
> Observed the following in recent Jenkins build
> (https://builds.apache.org/job/HBase-TRUNK/6732/console):
> {code}
> testWALfencingWithoutWALRolling(org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures)
> Time elapsed: 9.938 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: failed to create file
> /user/jenkins/test-data/0d9e3047-6bb1-4219-9ed2-5b9884176321/MasterProcWALs/state-0002.log
> for DFSClient_NONMAPREDUCE_-966558185_1 for client 127.0.0.1 because current
> leaseholder is trying to recreate file.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2589)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2386)
> {code}
> When file creation fails (e.g. due to RemoteException), we should handle the
> exception by returning false.
> Similar handling can be applied to failure in writing header.
> Thanks to [~mbertozzi] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700893#comment-14700893
]
Heng Chen commented on HBASE-14227:
---
I see...So i think if columnFamily is specified in {{Admin.compact}}, we
can do mob compaction if the cf is mob one.
And if columnFamily is not specified, we can add another interface like
{{Admin.compact(final TableName tableName, final boolean isMob)}} just as
[~anoop.hbase] said.
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Blocker
> Fix For: 2.0.0
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14078) improve error message when HMaster can't bind to port
[
https://issues.apache.org/jira/browse/HBASE-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700908#comment-14700908
]
Hadoop QA commented on HBASE-14078:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750945/hbase-14708-v3.patch
against master branch at commit 395ec5a9bb48324a8b7dd61790a954a2998a8f80.
ATTACHMENT ID: 12750945
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestAcidGuarantees
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
{color:red}-1 core zombie tests{color}. There are 9 zombie test(s):
at
org.apache.hadoop.hbase.client.TestFromClientSide.testKeepDeletedCells(TestFromClientSide.java:191)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15134//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15134//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15134//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15134//console
This message is automatically generated.
> improve error message when HMaster can't bind to port
> -
>
> Key: HBASE-14078
> URL: https://issues.apache.org/jira/browse/HBASE-14078
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Matt Warhaftig
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: hbase-14078_post_stack.txt, hbase-14708-v1.patch,
> hbase-14708-v2.patch, hbase-14708-v3.patch
>
>
> When the master fails to start becahse hbase.master.port is already taken,
> the log messages could make it easier to tell.
> {quote}
> 2015-07-14 13:10:02,667 INFO [main] regionserver.RSRpcServices:
> master/master01.example.com/10.20.188.121:16000 server-side HConnection
> retries=350
> 2015-07-14 13:10:02,879 INFO [main] ipc.SimpleRpcScheduler: Using deadline
> as user call queue, count=3
> 2015-07-14 13:10:02,895 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2258)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:234)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2272)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apac
[jira] [Updated] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-14210: -- Attachment: HBASE-14210-v3.patch > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-v1.patch, HBASE-14210-v2.patch, > HBASE-14210-v3.patch, HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700932#comment-14700932 ] Ashish Singhi commented on HBASE-14210: --- Updated patch to use user.getShortName in the fail message and renamed the variables of kind permsU2_GUandOwner as per Ted's suggestion. > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-v1.patch, HBASE-14210-v2.patch, > HBASE-14210-v3.patch, HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module
[
https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700936#comment-14700936
]
Hadoop QA commented on HBASE-14181:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750939/HBASE-14181.5.patch
against master branch at commit 395ec5a9bb48324a8b7dd61790a954a2998a8f80.
ATTACHMENT ID: 12750939
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1860 checkstyle errors (more than the master's current 1852 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ScanRange scanRange = new
ScanRange(rowRange.getStopRow().toByteArray(), rowRange.getStopRowInclusive(),
+ val schemaMappingDefinition:java.util.HashMap[String,
SchemaQualifierDefinition],
+requiredQualifierDefinitionArray.foreach( d =>
get.addColumn(d.columnFamilyBytes, d.qualifierBytes))
+requiredQualifierDefinitionArray.foreach( d =>
scan.addColumn(d.columnFamilyBytes, d.qualifierBytes))
+ requiredQualifierDefinitionArray.foreach( d =>
scan.addColumn(d.columnFamilyBytes, d.qualifierBytes))
+ var points:mutable.MutableList[Array[Byte]] = new
mutable.MutableList[Array[Byte]](),
+ var ranges:mutable.MutableList[ScanRange] = new
mutable.MutableList[ScanRange]() ) extends Serializable {
+ requiredQualifierDefinitionArray:
mutable.MutableList[SchemaQualifierDefinition]):Unit = {
+ Map("hbase.columns.mapping" -> "KEY_FIELD STRING :key, A_FIELD STRING
c:a, B_FIELD STRING c:b,",
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
org.apache.hadoop.hbase.TestStochasticBalancerJmxMetrics
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot
org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient
org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient
org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
org.apache.hadoop.hbase.client.TestScannersFromClientSide
org.apache.hadoop.hbase.snapshot.TestExportSnapshot
org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15137//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15137//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15137//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15137//console
This message is automatically generated.
> Add Spark DataFrame DataSource to HBase-Spark Module
>
>
> Key: HBASE-14181
> URL: https://issues.apache.org/jira/browse/HBASE-14181
> Project: HBase
> Issue Type: New Feature
> Components: spark
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Attachments: HBASE-14181.1.patch, HBASE-14181.2.patch,
> HBASE-14181.3.patch, HBASE-14181.4.patch, HBASE-141
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700939#comment-14700939 ] Jingcheng Du commented on HBASE-14227: -- I mean we'd better not compact the normal and mob cf together. I have a way to fold the compactMob into the existing APIs. As you know, the mob is stored in a dummy mob region (with a constant name). We can use the exiting APIs compactRegion and majorCompactRegion to do this. # If the region name is a mob region name, we do compaction of mob. # Otherwise do normal compaction. How about this? > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700941#comment-14700941 ] Anoop Sam John commented on HBASE-14227: This will work for the APIs which takes region name too. WHat abt API which takes only table name? Expectation is compact all regions for it. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700942#comment-14700942 ] Jingcheng Du commented on HBASE-14227: -- Wait... We still need table name and cf name in compactRegion, so this approach doesn't work. Sorry. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700944#comment-14700944 ] Anoop Sam John commented on HBASE-14210: +1 > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-v1.patch, HBASE-14210-v2.patch, > HBASE-14210-v3.patch, HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700956#comment-14700956 ] Srikanth Srungarapu commented on HBASE-14210: - +1 lgtm. Thanks, Ashish! [~te...@apache.org] Are you planning to commit the patch? > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-v1.patch, HBASE-14210-v2.patch, > HBASE-14210-v3.patch, HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700961#comment-14700961 ] ramkrishna.s.vasudevan commented on HBASE-14210: +1 on patch. > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-v1.patch, HBASE-14210-v2.patch, > HBASE-14210-v3.patch, HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700962#comment-14700962 ] Heng Chen commented on HBASE-14227: --- I have a question: Is there any need to add interface compact for MOB at table and region level? If user want to compact MOB CF, just specify it at CF Level. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14231) Javadoc warning in ProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-14231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14231: --- Resolution: Duplicate Status: Resolved (was: Patch Available) HBASE-13127 fixes this as per Stack's comment. > Javadoc warning in ProtobufUtil > --- > > Key: HBASE-14231 > URL: https://issues.apache.org/jira/browse/HBASE-14231 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Trivial > Attachments: HBASE-14231.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-14210: -- Attachment: HBASE-14210-branch-1.patch > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-branch-1.patch, HBASE-14210-v1.patch, > HBASE-14210-v2.patch, HBASE-14210-v3.patch, HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security
[
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700990#comment-14700990
]
Hudson commented on HBASE-14122:
FAILURE: Integrated in HBase-1.3 #113 (See
[https://builds.apache.org/job/HBase-1.3/113/])
HBASE-14122 Client API for determining if server side supports cell level
security - ADDENDUM for javadoc fix (enis: rev
c619e0c76ff5551552848e272952efbda4270203)
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
> Client API for determining if server side supports cell level security
> --
>
> Key: HBASE-14122
> URL: https://issues.apache.org/jira/browse/HBASE-14122
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0
>
> Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch,
> HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch,
> HBASE-14122.patch, HBASE-14122.patch
>
>
> Add a client API for determining if the server side supports cell level
> security.
> Ask the master, assuming as we do in many other instances that the master and
> regionservers all have a consistent view of site configuration.
> Return {{true}} if all features required for cell level security are present,
> {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master
> does not have support for the RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14203) remove duplicate code getTableDescriptor in HTable
[
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700989#comment-14700989
]
Hudson commented on HBASE-14203:
FAILURE: Integrated in HBase-1.3 #113 (See
[https://builds.apache.org/job/HBase-1.3/113/])
HBASE-14203 remove duplicate code getTableDescriptor in HTable (Heng Chen)
(enis: rev c9b3d837a0ca4677e9f86f35ef78b6cb4710527b)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
> remove duplicate code getTableDescriptor in HTable
> --
>
> Key: HBASE-14203
> URL: https://issues.apache.org/jira/browse/HBASE-14203
> Project: HBase
> Issue Type: Improvement
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14203.patch, HBASE-14203_v2.patch,
> HBASE-14203_v3.patch, HBASE-14203_v4.patch, HBASE-14203_v5.patch
>
>
> As TODO in comment said,
> {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}.
> remove the duplicate code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701014#comment-14701014 ] Jingcheng Du commented on HBASE-14227: -- No need for region level, just need to tell table and CF name. The thing is we don't want to do the compaction in store files and mob files together. The ref cells of mob are stored in store file in hbase, and the real mob data are stored in mob files outside hbase directory. The hbase compaction only needs to compact the ref cells, and mob compaction takes care of mob files. The impact of the number of mob files is not significant in read, usually they are deleted when expired. Sometimes the compactions to them are not necessary. We have two approaches for this. # Use a special name format in the cf name, if we want to do the mobCompact, we can pass in a special cf name, for instance cf$MOB. # Add two new APIs with one additional parameter (enum or boolean) to switch the mob compaction and normal compaction. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701027#comment-14701027
]
Heng Chen commented on HBASE-14227:
---
{quote}
1. Use a special name format in the cf name, if we want to do the mobCompact,
we can pass in a special cf name, for instance cf$MOB.
{quote}
There is no need to do this. If CF is passed in, we can check schema option to
decide whether the CF is MOB or not.
The question is when compact on table or region Level, shall we do MOB
compaction?
If we do, the function {{Admin.compact(final TableName tableName)}} need one
additional parameter to switch the mob compaction and normal compaction as
[~jingcheng...@intel.com] said.
If we don't, there is no need to change {{Admin.compact(final TableName
tableName)}}
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Blocker
> Fix For: 2.0.0
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14237) Meta region may be onlined on multi regonservers for bugs of assigning meta
Liu Shaohui created HBASE-14237: --- Summary: Meta region may be onlined on multi regonservers for bugs of assigning meta Key: HBASE-14237 URL: https://issues.apache.org/jira/browse/HBASE-14237 Project: HBase Issue Type: Bug Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Critical When a regionserver failed to open the meta region and crash after setting the RS_ZK_REGION_FAILED_OPEN state of meta region in zookeeper, the master will handle the event of RS_ZK_REGION_FAILED_OPEN and try to assign the meta region again in AssignmentManager#handleRegion. But at the same time, the master will handle the regionserver expired event and start a MetaServerShutdownHandler for the regionserver, because the servername of regionserver is same as the servername of the unassigned node of meta region. In the MetaServerShutdownHandler, the meta region may be assigned for second time. [~heliangliang] We have encountered this problem in our production cluster which resulted in inconsistency of region location in meta table. You can see the log from the attachment. The code of AssignmentManager is so complex and I have not get a solution to fix this problem. Could someone kindly help to give some suggestions? Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14237) Meta region may be onlined on multi regonservers for bugs of assigning meta
[ https://issues.apache.org/jira/browse/HBASE-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-14237: Attachment: meta.log RegionServer log for this issue > Meta region may be onlined on multi regonservers for bugs of assigning meta > --- > > Key: HBASE-14237 > URL: https://issues.apache.org/jira/browse/HBASE-14237 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Critical > Attachments: meta.log > > > When a regionserver failed to open the meta region and crash after setting > the RS_ZK_REGION_FAILED_OPEN state of meta region in zookeeper, the master > will handle the event of RS_ZK_REGION_FAILED_OPEN and try to assign the meta > region again in AssignmentManager#handleRegion. But at the same time, the > master will handle the regionserver expired event and start a > MetaServerShutdownHandler for the regionserver, because the servername of > regionserver is same as the servername of the unassigned node of meta region. > In the MetaServerShutdownHandler, the meta region may be assigned for second > time. > [~heliangliang] > We have encountered this problem in our production cluster which resulted in > inconsistency of region location in meta table. You can see the log from the > attachment. > The code of AssignmentManager is so complex and I have not get a solution to > fix this problem. Could someone kindly help to give some suggestions? Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701043#comment-14701043 ] Jingcheng Du commented on HBASE-14227: -- bq. There is no need to do this. If CF is passed in, we can check schema option to decide whether the CF is MOB or not Compacting the store files with ref cells and mob files are different. But they share the same CF. We want to separate the compactions, and we have to separate these operations in methods. If only with CF, we only know the cf is a mob-enable cf, but we don't know what kind of compaction is needed. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected
[
https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-13480:
-
Attachment: HBASE-13480.patch
Upload the first patch, No new unit tests are included.
Thanks for your comments.
> ShortCircuitConnection doesn't short-circuit all calls as expected
> --
>
> Key: HBASE-13480
> URL: https://issues.apache.org/jira/browse/HBASE-13480
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0
>Reporter: Josh Elser
>Assignee: Jingcheng Du
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: HBASE-13480.patch
>
>
> Noticed the following situation in debugging unexpected unit tests failures
> in HBASE-13351.
> {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName,
> AdminService.BlockingInterface, ClientService.BlockingInterface)}} is
> intended to avoid the extra RPC by calling the server's instantiation of the
> protobuf rpc stub directly for the AdminService and ClientService.
> The problem is that this is insufficient to actually avoid extra "remote"
> RPCs as all other calls to the Connection are routed to a "real" Connection
> instance. As such, any object created by the "real" Connection (such as an
> HTable) will use the real Connection, not the SSC.
> The end result is that
> {{MasterRpcService#reportRegionStateTransition(RpcController,
> ReportRegionStateTransitionRequest)}} will make additional "remote" RPCs over
> what it thinks is an SSC through a {{Get}} on {{HTable}} which was
> constructed using the SSC, but the {{Get}} itself will use the underlying
> real Connection instead of the SSC. With insufficiently sized thread pools,
> this has been observed to result in RPC deadlock in the HMaster where an RPC
> attempts to make another RPC but there are no more threads available to
> service the second RPC so the first RPC blocks indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701052#comment-14701052
]
Heng Chen commented on HBASE-14227:
---
{quote}
Compacting the store files with ref cells and mob files are different. But they
share the same CF. We want to separate the compactions, and we have to separate
these operations in methods. If only with CF, we only know the cf is a
mob-enable cf, but we don't know what kind of compaction is needed.
{quote}
I see... I agree with you. we can use special name to notify us do MOB
compaction.
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Blocker
> Fix For: 2.0.0
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701054#comment-14701054 ] Jingcheng Du commented on HBASE-14227: -- Thanks [~chenheng]! Let's see if any more comments on the first approach (special name format which adds limit to the cf name in that method). > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-14210: -- Attachment: HBASE-14210-0.98.patch > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-0.98.patch, HBASE-14210-branch-1.patch, > HBASE-14210-v1.patch, HBASE-14210-v2.patch, HBASE-14210-v3.patch, > HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected
[
https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-13480:
-
Status: Patch Available (was: Open)
> ShortCircuitConnection doesn't short-circuit all calls as expected
> --
>
> Key: HBASE-13480
> URL: https://issues.apache.org/jira/browse/HBASE-13480
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.1.0, 1.0.0, 2.0.0
>Reporter: Josh Elser
>Assignee: Jingcheng Du
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: HBASE-13480.patch
>
>
> Noticed the following situation in debugging unexpected unit tests failures
> in HBASE-13351.
> {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName,
> AdminService.BlockingInterface, ClientService.BlockingInterface)}} is
> intended to avoid the extra RPC by calling the server's instantiation of the
> protobuf rpc stub directly for the AdminService and ClientService.
> The problem is that this is insufficient to actually avoid extra "remote"
> RPCs as all other calls to the Connection are routed to a "real" Connection
> instance. As such, any object created by the "real" Connection (such as an
> HTable) will use the real Connection, not the SSC.
> The end result is that
> {{MasterRpcService#reportRegionStateTransition(RpcController,
> ReportRegionStateTransitionRequest)}} will make additional "remote" RPCs over
> what it thinks is an SSC through a {{Get}} on {{HTable}} which was
> constructed using the SSC, but the {{Get}} itself will use the underlying
> real Connection instead of the SSC. With insufficiently sized thread pools,
> this has been observed to result in RPC deadlock in the HMaster where an RPC
> attempts to make another RPC but there are no more threads available to
> service the second RPC so the first RPC blocks indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701104#comment-14701104
]
Jingcheng Du commented on HBASE-14227:
--
Go through the code, I think we can fold these mob APIs into {{void
compactRegion(final byte[] regionName, final byte[] columnFamily)}} and {{void
majorCompactRegion(final byte[] regionName, final byte[] columnFamily)}}.
# We could know whether it's a mob region by region name.
# Issue the compaction from admin by passing region name and column
name(optional).
I think this can work.
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Priority: Blocker
> Fix For: 2.0.0
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected
[
https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701108#comment-14701108
]
Jingcheng Du commented on HBASE-13480:
--
The patch is only for master. I will provide patches for other branches after
the review is finished.
> ShortCircuitConnection doesn't short-circuit all calls as expected
> --
>
> Key: HBASE-13480
> URL: https://issues.apache.org/jira/browse/HBASE-13480
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0
>Reporter: Josh Elser
>Assignee: Jingcheng Du
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: HBASE-13480.patch
>
>
> Noticed the following situation in debugging unexpected unit tests failures
> in HBASE-13351.
> {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName,
> AdminService.BlockingInterface, ClientService.BlockingInterface)}} is
> intended to avoid the extra RPC by calling the server's instantiation of the
> protobuf rpc stub directly for the AdminService and ClientService.
> The problem is that this is insufficient to actually avoid extra "remote"
> RPCs as all other calls to the Connection are routed to a "real" Connection
> instance. As such, any object created by the "real" Connection (such as an
> HTable) will use the real Connection, not the SSC.
> The end result is that
> {{MasterRpcService#reportRegionStateTransition(RpcController,
> ReportRegionStateTransitionRequest)}} will make additional "remote" RPCs over
> what it thinks is an SSC through a {{Get}} on {{HTable}} which was
> constructed using the SSC, but the {{Get}} itself will use the underlying
> real Connection instead of the SSC. With insufficiently sized thread pools,
> this has been observed to result in RPC deadlock in the HMaster where an RPC
> attempts to make another RPC but there are no more threads available to
> service the second RPC so the first RPC blocks indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14193) Remove support for direct upgrade from pre-0.96 versions
[ https://issues.apache.org/jira/browse/HBASE-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-14193: - Status: Open (was: Patch Available) > Remove support for direct upgrade from pre-0.96 versions > > > Key: HBASE-14193 > URL: https://issues.apache.org/jira/browse/HBASE-14193 > Project: HBase > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14193.patch > > > As discussed on the mailing list this will remove all support for upgrades > from pre-0.96 versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14170) [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address already in use.
[
https://issues.apache.org/jira/browse/HBASE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701141#comment-14701141
]
sunhaitao commented on HBASE-14170:
---
the hang comes from Jetty server, the jetty server is not handling the port
already bound exception
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> -
>
> Key: HBASE-14170
> URL: https://issues.apache.org/jira/browse/HBASE-14170
> Project: HBase
> Issue Type: Bug
> Components: REST
>Reporter: Y. SREENIVASULU REDDY
> Fix For: 2.0.0, 1.2.0, 1.0.3
>
>
> [HBase Rest] RESTServer is not shutting down if "hbase.rest.port" Address
> already in use.
> If "hbase.rest.port" Address already in use, RESTServer should shutdown,
> with out this "hbase.rest.port" we cant perform any operations on
> RESTServer. Then there is no use of running RESTServer process.
> {code}
> 2015-07-30 11:49:48,273 WARN [main] mortbay.log: failed
> SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already
> in use
> 2015-07-30 11:49:48,274 WARN [main] mortbay.log: failed Server@563f38c4:
> java.net.BindException: Address already in use
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14193) Remove support for direct upgrade from pre-0.96 versions
[ https://issues.apache.org/jira/browse/HBASE-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-14193: - Status: Patch Available (was: Open) > Remove support for direct upgrade from pre-0.96 versions > > > Key: HBASE-14193 > URL: https://issues.apache.org/jira/browse/HBASE-14193 > Project: HBase > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14193-v1.patch, HBASE-14193.patch > > > As discussed on the mailing list this will remove all support for upgrades > from pre-0.96 versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14193) Remove support for direct upgrade from pre-0.96 versions
[ https://issues.apache.org/jira/browse/HBASE-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated HBASE-14193: - Attachment: HBASE-14193-v1.patch New patch that addresses the comments. It does not add a warning to the windows command (hbase.cmd) as I don't know batch script stuff and it's a different pattern from linux. I think that's alright though as the message is just an added bonus. > Remove support for direct upgrade from pre-0.96 versions > > > Key: HBASE-14193 > URL: https://issues.apache.org/jira/browse/HBASE-14193 > Project: HBase > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14193-v1.patch, HBASE-14193.patch > > > As discussed on the mailing list this will remove all support for upgrades > from pre-0.96 versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[ https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunyerui updated HBASE-14229: - Attachment: HBASE-14229-branch-1.patch HBASE-14229-0.98.patch Here is the patches, for 0.98 and branch-1, generated by git format-patch. In the patch for branch 0.98, in order to get the committed file list after StoreFlushContext.commit(), a new method getCommittedFiles() is added into Interface StoreFlushContext. I'm not sure whether it's a good idea, but I didn't figured out another way to do this. Any comments on this will be appreciated. > Flushing canceled by coprocessor still leads to memstoreSize set down > - > > Key: HBASE-14229 > URL: https://issues.apache.org/jira/browse/HBASE-14229 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: sunyerui > Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1.patch > > > A Coprocessor override "public InternalScanner preFlush(final Store store, > final InternalScanner scanner)" and return NULL when calling this method, > will cancel flush request, leaving snapshot un-flushed, and no new storefile > created. But the HRegion.internalFlushCache still set down memstoreSize to 0 > by totalFlushableSize. > If there's no write requests anymore, the memstoreSize will remaining as 0, > and no more flush quests will be processed because of the checking of > memstoreSize.get() <=0 at the beginning of internalFlushCache. > This issue may not cause data loss, but it will confuse coprocessor users. If > we argree with this, I'll apply a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[
https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701192#comment-14701192
]
Hadoop QA commented on HBASE-14210:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12750993/HBASE-14210-branch-1.patch
against branch-1 branch at commit 395ec5a9bb48324a8b7dd61790a954a2998a8f80.
ATTACHMENT ID: 12750993
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.mapreduce.TestHashTable
org.apache.hadoop.hbase.mapreduce.TestHRegionPartitioner
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad
org.apache.hadoop.hbase.coprocessor.TestHTableWrapper
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs
org.apache.hadoop.hbase.mapreduce.TestCellCounter
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.mapreduce.TestCopyTable
org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
org.apache.hadoop.hbase.mapreduce.TestWALPlayer
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
org.apache.hadoop.hbase.mapreduce.TestRowCounter
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
org.apache.hadoop.hbase.mapreduce.TestSyncTable
org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy
org.apache.hadoop.hbase.mapreduce.TestTableInputFormat
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery
org.apache.hadoop.hbase.TestIOFencing
org.apache.hadoop.hbase.regionserver.TestMajorCompaction
org.apache.hadoop.hbase.regionserver.TestScannerRetriableFailure
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes
{color:red}-1 core zombie tests{color}. There are 15 zombie test(s):
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testFixAssignmentsWhenMETAinTransition(TestHBaseFsck.java:283)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testOrphanedTableZNode(TestHBaseFsck.java:2497)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15140//testRepor
[jira] [Updated] (HBASE-13996) Add write sniffing in canary
[ https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-13996: Attachment: HBASE-13996-v002.diff Update for [~stack]'s review~ Changes: - Add writeSniffing option - Change the default value of regions per regionserver to 1 - Add enabling write sniffing doc > Add write sniffing in canary > > > Key: HBASE-13996 > URL: https://issues.apache.org/jira/browse/HBASE-13996 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 0.98.13, 1.1.0.1 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Fix For: 2.0.0, 1.3.0, 0.98.15 > > Attachments: HBASE-13996-v001.diff, HBASE-13996-v002.diff > > > Currently the canary tool only sniff the read operations, it's hard to > finding the problem in write path. > To support the write sniffing, we create a system table named '_canary_' in > the canary tool. And the tool will make sure that the region number is large > than the number of the regionserver and the regions will be distributed onto > all regionservers. > Periodically, the tool will put data to these regions to calculate the write > availability of HBase and send alerts if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13996) Add write sniffing in canary
[ https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701231#comment-14701231 ] Liu Shaohui commented on HBASE-13996: - [~stack] [~apurtell] Could you help to review the patch? Sorry for leaving the issue here so long. > Add write sniffing in canary > > > Key: HBASE-13996 > URL: https://issues.apache.org/jira/browse/HBASE-13996 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 0.98.13, 1.1.0.1 >Reporter: Liu Shaohui >Assignee: Liu Shaohui > Fix For: 2.0.0, 1.3.0, 0.98.15 > > Attachments: HBASE-13996-v001.diff, HBASE-13996-v002.diff > > > Currently the canary tool only sniff the read operations, it's hard to > finding the problem in write path. > To support the write sniffing, we create a system table named '_canary_' in > the canary tool. And the tool will make sure that the region number is large > than the number of the regionserver and the regions will be distributed onto > all regionservers. > Periodically, the tool will put data to these regions to calculate the write > availability of HBase and send alerts if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701241#comment-14701241 ] Ted Yu commented on HBASE-14210: Srikanth: Please go ahead with commit. > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-0.98.patch, HBASE-14210-branch-1.patch, > HBASE-14210-v1.patch, HBASE-14210-v2.patch, HBASE-14210-v3.patch, > HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module
[ https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated HBASE-14181: Attachment: HBASE-14181.6.patch Fixed long lines and warners > Add Spark DataFrame DataSource to HBase-Spark Module > > > Key: HBASE-14181 > URL: https://issues.apache.org/jira/browse/HBASE-14181 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Attachments: HBASE-14181.1.patch, HBASE-14181.2.patch, > HBASE-14181.3.patch, HBASE-14181.4.patch, HBASE-14181.5.patch, > HBASE-14181.6.patch > > > Build a RelationProvider for HBase-Spark Module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14122) Client API for determining if server side supports cell level security
[
https://issues.apache.org/jira/browse/HBASE-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701259#comment-14701259
]
Hudson commented on HBASE-14122:
FAILURE: Integrated in HBase-1.2 #116 (See
[https://builds.apache.org/job/HBase-1.2/116/])
HBASE-14122 Client API for determining if server side supports cell level
security - ADDENDUM for javadoc fix (enis: rev
05328c7a46eb52096d55ee0ae86f1bc99000b692)
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
> Client API for determining if server side supports cell level security
> --
>
> Key: HBASE-14122
> URL: https://issues.apache.org/jira/browse/HBASE-14122
> Project: HBase
> Issue Type: Improvement
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 2.0.0, 0.98.14, 1.2.0, 1.3.0
>
> Attachments: HBASE-14122-0.98.patch, HBASE-14122-0.98.patch,
> HBASE-14122-branch-1.patch, HBASE-14122-branch-1.patch, HBASE-14122.patch,
> HBASE-14122.patch, HBASE-14122.patch
>
>
> Add a client API for determining if the server side supports cell level
> security.
> Ask the master, assuming as we do in many other instances that the master and
> regionservers all have a consistent view of site configuration.
> Return {{true}} if all features required for cell level security are present,
> {{false}} otherwise, or throw {{UnsupportedOperationException}} if the master
> does not have support for the RPC call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14203) remove duplicate code getTableDescriptor in HTable
[
https://issues.apache.org/jira/browse/HBASE-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701258#comment-14701258
]
Hudson commented on HBASE-14203:
FAILURE: Integrated in HBase-1.2 #116 (See
[https://builds.apache.org/job/HBase-1.2/116/])
HBASE-14203 remove duplicate code getTableDescriptor in HTable (Heng Chen)
(enis: rev 158839552fb765e3e330a3a905511c22eec6b2ce)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
> remove duplicate code getTableDescriptor in HTable
> --
>
> Key: HBASE-14203
> URL: https://issues.apache.org/jira/browse/HBASE-14203
> Project: HBase
> Issue Type: Improvement
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14203.patch, HBASE-14203_v2.patch,
> HBASE-14203_v3.patch, HBASE-14203_v4.patch, HBASE-14203_v5.patch
>
>
> As TODO in comment said,
> {{HTable.getTableDescriptor}} is same as {{HAdmin.getTableDescriptor}}.
> remove the duplicate code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701284#comment-14701284 ] Dave Latham commented on HBASE-14227: - I'm not a fan of passing behavior arguments as special in band names for something like a column family. It leads to worse documentation, user surprises, and potentially even conflicts if someone already has such a column family. Since MOB compaction actually is a separate thing, it should be reflected in the API by either another parameter or new API methods (though they should be minimized). But just my two cents from an interested user. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14193) Remove support for direct upgrade from pre-0.96 versions
[ https://issues.apache.org/jira/browse/HBASE-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701290#comment-14701290 ] Dave Latham commented on HBASE-14193: - +1 (not a committer) > Remove support for direct upgrade from pre-0.96 versions > > > Key: HBASE-14193 > URL: https://issues.apache.org/jira/browse/HBASE-14193 > Project: HBase > Issue Type: Improvement >Reporter: Lars Francke >Assignee: Lars Francke >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14193-v1.patch, HBASE-14193.patch > > > As discussed on the mailing list this will remove all support for upgrades > from pre-0.96 versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[
https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701397#comment-14701397
]
Hadoop QA commented on HBASE-14210:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12751011/HBASE-14210-0.98.patch
against 0.98 branch at commit 395ec5a9bb48324a8b7dd61790a954a2998a8f80.
ATTACHMENT ID: 12751011
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
22 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestHBaseConfiguration
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.master.TestMasterShutdown.testMasterShutdownBeforeStartingAnyRegionServer(TestMasterShutdown.java:143)
at
org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testReadWriteSeqIdFiles(TestDistributedLogSplitting.java:1401)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15142//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15142//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15142//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15142//console
This message is automatically generated.
> Create test for cell level ACLs involving user group
>
>
> Key: HBASE-14210
> URL: https://issues.apache.org/jira/browse/HBASE-14210
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Attachments: HBASE-14210-0.98.patch, HBASE-14210-branch-1.patch,
> HBASE-14210-v1.patch, HBASE-14210-v2.patch, HBASE-14210-v3.patch,
> HBASE-14210.patch
>
>
> Currently we have TestCellACLs and TestCellACLWithMultipleVersions which
> exercise cell level ACLs for users.
> However, test for cell level ACLs involving user group is missing.
> This issue is to add such test(s)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13127) Add timeouts on all tests so less zombie sightings
[ https://issues.apache.org/jira/browse/HBASE-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13127: -- Attachment: 13127.alternate.v7.txt javadoc warning was fixed elsewhere. Purge that change from this patch. > Add timeouts on all tests so less zombie sightings > -- > > Key: HBASE-13127 > URL: https://issues.apache.org/jira/browse/HBASE-13127 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: stack >Assignee: stack > Attachments: 13127.alternate.txt, 13127.alternate.txt, > 13127.alternate.txt, 13127.alternate.txt, 13127.alternate.v2.txt, > 13127.alternate.v3.txt, 13127.alternate.v3.txt, 13127.alternate.v3.txt, > 13127.alternate.v3.txt, 13127.alternate.v4.txt, 13127.alternate.v5.txt, > 13127.alternate.v6.txt, 13127.alternate.v6.txt, 13127.alternate.v7.txt, > 13127.txt, 13127v2.txt > > > [~Apache9] and [~octo47] have been working hard at trying to get our builds > passing again. They are almost there. TRUNK just failed with a zombie > TestMasterObserver. Help the lads out by adding timeouts on all tests so less > zombie incidence... will help identify the frequent failing issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14234) Exception encountered in WALProcedureStore#rollWriter() should be properly handled
[
https://issues.apache.org/jira/browse/HBASE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14234:
---
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
1.2.0
2.0.0
Thanks for the review.
I ran the tests reported as zombie and they passed locally.
> Exception encountered in WALProcedureStore#rollWriter() should be properly
> handled
> --
>
> Key: HBASE-14234
> URL: https://issues.apache.org/jira/browse/HBASE-14234
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14234-v1.txt
>
>
> Observed the following in recent Jenkins build
> (https://builds.apache.org/job/HBase-TRUNK/6732/console):
> {code}
> testWALfencingWithoutWALRolling(org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures)
> Time elapsed: 9.938 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: failed to create file
> /user/jenkins/test-data/0d9e3047-6bb1-4219-9ed2-5b9884176321/MasterProcWALs/state-0002.log
> for DFSClient_NONMAPREDUCE_-966558185_1 for client 127.0.0.1 because current
> leaseholder is trying to recreate file.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2589)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2386)
> {code}
> When file creation fails (e.g. due to RemoteException), we should handle the
> exception by returning false.
> Similar handling can be applied to failure in writing header.
> Thanks to [~mbertozzi] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14232) Backwards compatiblity support for new MasterObserver APIs
[ https://issues.apache.org/jira/browse/HBASE-14232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701523#comment-14701523 ] Andrew Purtell commented on HBASE-14232: bq. We can generically do this for all types of CPs? So later on we can add hooks to CPs with out worry for BC. We already have some reusable support for this from when Sean updated the WALObserver. It's reasonable to aim for as much reuse as we can get when doing the work here. > Backwards compatiblity support for new MasterObserver APIs > -- > > Key: HBASE-14232 > URL: https://issues.apache.org/jira/browse/HBASE-14232 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell > Labels: hbase-6721 > > The group assignment changes introduce new methods to the MasterObserver > interface. This is a concern for things like Apache Phoenix. (See their > IndexMasterObserver, etc.) We can handle this by using compatibility helpers > that won't attempt to invoke the new APIs on MasterObservers that do not > implement them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11116) Allow ReplicationSink to write to tables when READONLY is set
[ https://issues.apache.org/jira/browse/HBASE-6?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6. Resolution: Won't Fix Sounds reasonable. Resolving > Allow ReplicationSink to write to tables when READONLY is set > - > > Key: HBASE-6 > URL: https://issues.apache.org/jira/browse/HBASE-6 > Project: HBase > Issue Type: New Feature > Components: regionserver, Replication >Reporter: Geoffrey Anderson > > A common practice with MySQL replication is to enable the read_only flag > (http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_read_only) > on slaves in order to disallow any user changes to a slave, while allowing > replication from the master to flow in. In HBase, enabling the READONLY > field on a table will cause ReplicationSink to fail to apply changes. It'd > be ideal to have the default behavior allow replication to continue going > through, a configurable option to influence this type of behavior, or perhaps > even a flag that manipulates a peer's state (e.g. via add_peer) that dictates > if replication can write to READONLY tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14227: --- Assignee: Heng Chen > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14224) Fix coprocessor handling of duplicate classes
[ https://issues.apache.org/jira/browse/HBASE-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14224: -- Attachment: 14224v2.txt v2 In HTD, add check in setValue that prevents adding coprocessor via this means that skirts checks for such as duplicate loadings. This is a change in (an exotic) behavior. Adds to HTD, addCoprocessorWithSpecStr for shell to call in place of setValue. Some small refactoring in HTD to avoid duplicated code. In HConstants, add doc around CP_HTD_ATTR_VALUE_PATTERN since it foregrounded by the changes in here. In CoprocessorHost, guard against double-loading a cp in current session. Add some tests. No shell test added because it has test to add cp already so at least we have not broken loading cp by shell > Fix coprocessor handling of duplicate classes > - > > Key: HBASE-14224 > URL: https://issues.apache.org/jira/browse/HBASE-14224 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.0.0, 1.0.1, 1.2.0, 1.1.1 >Reporter: Lars George >Assignee: stack >Priority: Critical > Attachments: 14224.txt, 14224v2.txt, problem.pdf > > > While discussing with [~misty] over on HBASE-13907 we noticed some > inconsistency when copros are loaded. Sometimes you can load them more than > once, sometimes you can not. Need to consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-14227: - Attachment: HBASE-14227.patch Upload the 1st patch for review. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701533#comment-14701533 ] Jingcheng Du commented on HBASE-14227: -- Oops, got race condition with the assign. Sorry for that. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701534#comment-14701534 ] Andrew Purtell commented on HBASE-14227: Agreed overloading existing admin methods with a new option with an additional parameter is a better compromise. What we are doing here is exposing more control over compaction IO scheduling. We should update the APIs with that in mind and not make it MOB specific. Rather than a boolean I'd suggest a set of flags... I guess the Java idiom is a List of Enum... for selecting what "types" of CFs to include in the compaction. Or, something more flexible (but crazier) like a Filter or Comparator. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701536#comment-14701536 ] Andrew Purtell commented on HBASE-14227: My bad Jingcheng. It should be assigned to whomever is doing the work. Feel free to change. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701535#comment-14701535
]
Jingcheng Du commented on HBASE-14227:
--
Understand your concerns.
So how about passing the mob region name to the existing Admin APIs for
compaction? The mob region name can be found by calling
{{HRegionInfo.getMobRegionName(TableName)}}.
> Fold special cased MOB APIs into existing APIs
> --
>
> Key: HBASE-14227
> URL: https://issues.apache.org/jira/browse/HBASE-14227
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.0.0
>Reporter: Andrew Purtell
>Assignee: Heng Chen
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-14227.patch
>
>
> There are a number of APIs that came in with MOB that are not new actions for
> HBase, simply new actions for a MOB implementation:
> - compactMob
> - compactMobs
> - majorCompactMob
> - majorCompactMobs
> - getMobCompactionState
> And in HBaseAdmin:
> - validateMobColumnFamily
> Remove these special cases from the Admin API where possible by folding them
> into existing APIs.
> We definitely don't need one method for a singleton and another for
> collections.
> Ideally we will not have any APIs named *Mob when finished, whether MOBs are
> in use on a table or not should be largely an internal detail. Exposing as
> schema option would be fine, this conforms to existing practice for other
> features.
> Marking critical because I think removing the *Mob special cased APIs should
> be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701541#comment-14701541 ] Andrew Purtell commented on HBASE-14227: I really don't like any *mob* API name of any kind. We can leak if a CF is a MOB through schema attributes, that seems fine. Otherwise I feel it is a layering violation. Our APIs shouldn't care what kind of CF is a CF at the level of method names. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14236) A bug in Reference Guide, Quick Start section
[ https://issues.apache.org/jira/browse/HBASE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701559#comment-14701559 ] stack commented on HBASE-14236: --- Can you suggest some text [~kramerli] Should we fix the advanced section rather than quick start? Idea is that quick start has minimal required config to get you going whereas by the time you are at advanced section, you have a bit of an idea as to what is going on. > A bug in Reference Guide, Quick Start section > - > > Key: HBASE-14236 > URL: https://issues.apache.org/jira/browse/HBASE-14236 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 1.0.1.1 > Environment: CentOS Linux release 7.1.1503 (Core) >Reporter: kramerli > Fix For: 1.0.1.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > This is an error on online documation. > (http://hbase.apache.org/book.html#_get_started_with_hbase ). > To locate the error you can go to > 2.2. Get Started with HBase > Example 2. Example hbase-site.xml for Standalone HBase > The config example here do not specify a port for the region server. So When > you go to the 2.4 Advanced - Fully Distributed > (http://hbase.apache.org/book.html#quickstart_fully_distributed). You will > hit error. > Because the region server will conflict with backup master or master. As they > use the same port. > People who read this section are always newbee, this will confuse them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701564#comment-14701564 ] Ted Yu commented on HBASE-14210: I ran TestCellACLs and TestCellACLWithMultipleVersions with patch for branch-1 which passed. The other test failures were not related to the patch since only the above tests were changed. > Create test for cell level ACLs involving user group > > > Key: HBASE-14210 > URL: https://issues.apache.org/jira/browse/HBASE-14210 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ashish Singhi > Attachments: HBASE-14210-0.98.patch, HBASE-14210-branch-1.patch, > HBASE-14210-v1.patch, HBASE-14210-v2.patch, HBASE-14210-v3.patch, > HBASE-14210.patch > > > Currently we have TestCellACLs and TestCellACLWithMultipleVersions which > exercise cell level ACLs for users. > However, test for cell level ACLs involving user group is missing. > This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701575#comment-14701575 ] Jingcheng Du commented on HBASE-14227: -- Thanks Andy for comments! How about folding the mob compaction into compactRegion/majorCompactRegion? This doesn't add additional words into names of methods. But users need to pass the mob region name (a dummy one) into these methods. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13480) ShortCircuitConnection doesn't short-circuit all calls as expected
[
https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701579#comment-14701579
]
Hadoop QA commented on HBASE-13480:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12751005/HBASE-13480.patch
against master branch at commit 395ec5a9bb48324a8b7dd61790a954a2998a8f80.
ATTACHMENT ID: 12751005
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.client.TestReplicaWithCluster.testReplicaAndReplication(TestReplicaWithCluster.java:300)
at
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testCloneSnapshotOfCloned(TestRestoreSnapshotFromClient.java:237)
at
org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient.testCloneSnapshot(TestMobCloneSnapshotFromClient.java:175)
at
org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient.testCloneSnapshotCrossNamespace(TestMobCloneSnapshotFromClient.java:189)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15141//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15141//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15141//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15141//console
This message is automatically generated.
> ShortCircuitConnection doesn't short-circuit all calls as expected
> --
>
> Key: HBASE-13480
> URL: https://issues.apache.org/jira/browse/HBASE-13480
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 1.0.0, 2.0.0, 1.1.0
>Reporter: Josh Elser
>Assignee: Jingcheng Du
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: HBASE-13480.patch
>
>
> Noticed the following situation in debugging unexpected unit tests failures
> in HBASE-13351.
> {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName,
> AdminService.BlockingInterface, ClientService.BlockingInterface)}} is
> intended to avoid the extra RPC by calling the server's instantiation of the
> protobuf rpc stub directly for the AdminService and ClientService.
> The problem is that this is insufficient to actually avoid extra "remote"
> RPCs as all other calls to the Connection are routed to a "real" Connection
> instance. As such, any object created by the "real" Connection (such as an
> HTable) will use the real Connection, not the SSC.
> The end result is that
> {{MasterRpcService#reportRegionStateTransition(RpcController,
> ReportRegionStateTransitionRequest)}} will make additional "remote" RPCs over
> what it thinks is an SSC through a {{Get}} on {{HTable}} which was
> constructed using the SSC, but the {{Get}} itself will use the underlying
> real Connection instead of the SSC. With insufficiently sized thread pools,
> this has been observed to result in RPC deadlock in the HMaster where an RPC
> attempts to make another RPC but there are no more threads available to
> service the second RPC so the first R
[jira] [Updated] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-14233:
--
Issue Type: Sub-task (was: Bug)
Parent: HBASE-14238
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14238) Branch-1.2 AM issues
Elliott Clark created HBASE-14238: - Summary: Branch-1.2 AM issues Key: HBASE-14238 URL: https://issues.apache.org/jira/browse/HBASE-14238 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 1.2.0 Reporter: Elliott Clark Priority: Blocker Fix For: 1.2.0 Parent jira for issues found through IT tests with Chaos monkey. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[
https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701587#comment-14701587
]
Elliott Clark commented on HBASE-14239:
---
RIT list contains:
{code}
Region 1588230740: hbase:meta,,1.1588230740 state=PENDING_CLOSE, ts=Tue Aug 18
03:48:30 PDT 2015 (22029s ago),
server=hbase498.ash1.facebook.com,16020,1439847225385
{code}
Then the RPC reader threads are blocked here:
{code}
Thread 29
(RpcServer.reader=5,bindAddress=hbasectrl023.ash1.facebook.com,port=16000):
State: WAITING
Blocked count: 13
Waited count: 14
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dc2455e
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:77)
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:197)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1856)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1753)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1612)
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1592)
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:856)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:641)
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:617)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
{code}
Priority handlers are all stuck here:
{code}
Thread 69 (PriorityRpcServer.handler=0,queue=0,port=16000):
State: TIMED_WAITING
Blocked count: 515
Waited count: 215347
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.waitUntilDone(AsyncProcess.java:1573)
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.waitUntilDone(AsyncProcess.java:1543)
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.processBatchCallback(ConnectionManager.java:2231)
org.apache.hadoop.hbase.util.MultiHConnection.processBatchCallback(MultiHConnection.java:124)
org.apache.hadoop.hbase.master.RegionStateStore.updateRegionState(RegionStateStore.java:244)
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:)
org.apache.hadoop.hbase.master.RegionStates.regionOnline(RegionStates.java:449)
org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:1460)
org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:3634)
org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:4263)
org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1339)
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
{code}
Assignment manager is here:
{code}
Thread 90156 (AM.-pool1-t403):
State: TIMED_WAITING
Blocked count: 302
Waited count: 206355
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.waitUntilDone(AsyncProcess.java:1573)
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.waitUntilDone(AsyncProcess.java:1543)
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.processBatchCallback(ConnectionManager.java:2231)
org.apache.hadoop.hbase.util.MultiHConnection.processBatchCallback(MultiHConnection.java:124)
org.apache.hadoop.hbase.master.RegionStateStore.updateRegionState(RegionStateStore.java:244)
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:)
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:427)
org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:385)
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager
[jira] [Commented] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701593#comment-14701593
]
Matteo Bertozzi commented on HBASE-14233:
-
[~eclark] proc-v2 is in 1.1 and I don't remember changes between 1.2 and 1.1.
and for sure nothing related to assignment or merging region, at the moment
proc-v2 is just for create/delete/modify tables. do you have more details on
your findings?
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14224) Fix coprocessor handling of duplicate classes
[ https://issues.apache.org/jira/browse/HBASE-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14224: -- Attachment: 14224v3.txt Address Andy's review from rb. > Fix coprocessor handling of duplicate classes > - > > Key: HBASE-14224 > URL: https://issues.apache.org/jira/browse/HBASE-14224 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.0.0, 1.0.1, 1.2.0, 1.1.1 >Reporter: Lars George >Assignee: stack >Priority: Critical > Attachments: 14224.txt, 14224v2.txt, 14224v3.txt, problem.pdf > > > While discussing with [~misty] over on HBASE-13907 we noticed some > inconsistency when copros are loaded. Sometimes you can load them more than > once, sometimes you can not. Need to consolidate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
Elliott Clark created HBASE-14239: - Summary: Branch-1.2 AM can get stuck when meta moves Key: HBASE-14239 URL: https://issues.apache.org/jira/browse/HBASE-14239 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14236) A bug in Reference Guide, Quick Start section
[ https://issues.apache.org/jira/browse/HBASE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701600#comment-14701600 ] kramerli commented on HBASE-14236: -- Thanks for the reply. Yes, quick start should has minimal required config. However the advanced section is also part of the quick start. http://hbase.apache.org/book.html#quickstart_fully_distributed Newbee who follow the instruction to setup the env and find there is fatal error will feel very frustrated. My suggestion is to add some text like this: """ Please be noted that in current Hbase version, the master or backup master server is also a type of region server. So when you decide to start the region server and backup master on the same node for example node-b.example.com. It will hit error as the port can be used for only one of them. To resolve this issue you can edit config/hbase-site.xml add below config hbase.regionserver.port 18020 hbase-default.xml This will force the region server to use some different port to avoid the conflict. """ I think the text can be added at the "Procedure: Prepare node-b and node-c". :) > A bug in Reference Guide, Quick Start section > - > > Key: HBASE-14236 > URL: https://issues.apache.org/jira/browse/HBASE-14236 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 1.0.1.1 > Environment: CentOS Linux release 7.1.1503 (Core) >Reporter: kramerli > Fix For: 1.0.1.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > This is an error on online documation. > (http://hbase.apache.org/book.html#_get_started_with_hbase ). > To locate the error you can go to > 2.2. Get Started with HBase > Example 2. Example hbase-site.xml for Standalone HBase > The config example here do not specify a port for the region server. So When > you go to the 2.4 Advanced - Fully Distributed > (http://hbase.apache.org/book.html#quickstart_fully_distributed). You will > hit error. > Because the region server will conflict with backup master or master. As they > use the same port. > People who read this section are always newbee, this will confuse them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701607#comment-14701607
]
Elliott Clark commented on HBASE-14233:
---
Didn't proc-v2 get assignment in 1.2?
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13996) Add write sniffing in canary
[
https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701605#comment-14701605
]
Andrew Purtell commented on HBASE-13996:
No problem at all [~liushaohui].
This is fine for when the canary is using the default canary table name
"hbase:canary". It's the explanation to the user that Stack asked for.
{code}
+} else if (tableName.equals(Canary.CANARY_TABLE_NAME)){
+description = "The hbase:canary table is used to sniff the write
availbility of each regionserver.";
{code}
However, I think the tool should allow for the user to give a different name
for the canary table. The default can be "hbase:canary" of course and nobody
would need to change it today. However I can see a future time when we could
have different resource allocation and admission control policies by namespace
and in that case a user might want to run a separate canary instance for the
different namespaces that are in use.
Since we are publishing write timing, should the size of the value being
written be configurable? This way a user can also watch for latency blips when
processing their use case's average or p95 or whatever value size. Should be
fine, we will hold down the total amount of data written because each iteration
of the canary will rewrite the same locations. Minor detail, feel free to
ignore.
Should this interval be configurable?
{code}
+// check canary distribution for every 10 minutes
{code}
Should the slop factor be configurable?
{code}
+ if (numberOfRegions < numberOfServers * regionsPerServer * 0.7
+ || numberOfRegions > numberOfServers * regionsPerServer * 1.5) {
{code}
Why 30% less but 50% more?
Ah, even better, we are setting TTLs on the canary family data. Great. Should
this be adjustable? Perhaps with a command line option.
{code}
+ family.setTimeToLive(24 * 60 * 60 * 1000);
{code}
Should be good feedback for now (smile)
> Add write sniffing in canary
>
>
> Key: HBASE-13996
> URL: https://issues.apache.org/jira/browse/HBASE-13996
> Project: HBase
> Issue Type: Improvement
> Components: canary
>Affects Versions: 0.98.13, 1.1.0.1
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
> Fix For: 2.0.0, 1.3.0, 0.98.15
>
> Attachments: HBASE-13996-v001.diff, HBASE-13996-v002.diff
>
>
> Currently the canary tool only sniff the read operations, it's hard to
> finding the problem in write path.
> To support the write sniffing, we create a system table named '_canary_' in
> the canary tool. And the tool will make sure that the region number is large
> than the number of the regionserver and the regions will be distributed onto
> all regionservers.
> Periodically, the tool will put data to these regions to calculate the write
> availability of HBase and send alerts if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14239: -- Description: When regions are moving master can get totally stuck trying to talk to meta. > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Reporter: Elliott Clark > Fix For: 1.2.0 > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701612#comment-14701612
]
Matteo Bertozzi commented on HBASE-14233:
-
[~eclark] nah, nothing was done for 1.2. we may have it for 1.3. we will talk
about it at the meetup on the 26th. but for now there is no code for AM
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-14233:
--
Component/s: (was: proc-v2)
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701617#comment-14701617
]
Elliott Clark commented on HBASE-14233:
---
H there goes my only lead then.
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
Nick Dimiduk created HBASE-14240:
Summary: Unactionable WARN messages in wal.ProcedureWALFormatReader
Key: HBASE-14240
URL: https://issues.apache.org/jira/browse/HBASE-14240
Project: HBase
Issue Type: Bug
Components: master, proc-v2
Affects Versions: 1.1.2
Reporter: Nick Dimiduk
Priority: Minor
While testing 1.1.2RC0, I notice the following messages in my master log.
{noformat}
2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
procedure2.ProcedureExecutor: Starting procedure executor threads=5
2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
util.FSHDFSUtils: Recovering lease on dfs file
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
util.FSHDFSUtils: recoverLease=true, attempt=0 on
file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
after 10ms
2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Opening state-log:
FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
isDirectory=false; length=9; replication=3; blocksize=134217728;
modification_time=1439858084806; access_time=1439857575101; owner=hbase;
group=hdfs; permission=rw-r--r--; isSymlink=false}
2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Unable to read tracker for
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
- Missing trailer: size=9 startPos=9
2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 2
2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Lease acquired for flushLogId: 2
2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: No active entry found in state log
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
removing it
2015-08-18 00:42:59,711 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Remove log:
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
{noformat}
These log lines are raised immediately after performing an in-place master
upgrade from 0.98.0. Master initialized successfully and with no further errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-14240:
-
Attachment: ndimiduk-rc-dist-1.tgz
Attaching complete master logs for reference.
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
> 2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
> 2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: Recovering lease on dfs file
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> 2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: recoverLease=true, attempt=0 on
> file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> after 10ms
> 2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Opening state-log:
> FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
> isDirectory=false; length=9; replication=3; blocksize=134217728;
> modification_time=1439858084806; access_time=1439857575101; owner=hbase;
> group=hdfs; permission=rw-r--r--; isSymlink=false}
> 2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Unable to read tracker for
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> - Missing trailer: size=9 startPos=9
> 2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Roll new state log: 2
> 2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Lease acquired for flushLogId: 2
> 2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
> 2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: No active entry found in state log
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
> removing it
> 2015-08-18 00:42:59,711 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Remove log:
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> {noformat}
> These log lines are raised immediately after performing an in-place master
> upgrade from 0.98.0. Master initialized successfully and with no further
> errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13376) Improvements to Stochastic load balancer
[ https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Ayyalasomayajula updated HBASE-13376: - Attachment: HBASE-13376_v3_0.98.patch Patch with review comments addressed. > Improvements to Stochastic load balancer > > > Key: HBASE-13376 > URL: https://issues.apache.org/jira/browse/HBASE-13376 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.0.0, 0.98.12 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: 13376-v2.txt, 13376-v5.patch, 13376_4.patch, > HBASE-13376.patch, HBASE-13376_0.98.txt, HBASE-13376_0.98_v2.patch, > HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, > HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_3.patch, > HBASE-13376_3.patch, HBASE-13376_4.patch, HBASE-13376_5_branch-1.patch, > HBASE-13376_6_branch-1.patch, HBASE-13376_98.patch, > HBASE-13376_branch-1.patch, HBASE-13376_v3_0.98.patch > > > There are two things this jira tries to address: > 1. The locality picker in the stochastic balancer does not pick regions with > least locality as candidates for swap/move. So when any user configures > locality cost in the configs, the balancer does not always seems to move > regions with bad locality. > 2. When a cluster has equal number of loaded regions, it always picks the > first one. It should pick a random region on one of the equally loaded > servers. This improves a chance of finding a good candidate, when load picker > is invoked several times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701650#comment-14701650
]
Matteo Bertozzi commented on HBASE-14240:
-
is the master shutdown clean? i'm not able to understand it from the logs.
in theory on a clean master shutdown the trailer should be written to the wal
and the file closed. which seems not to be happened here. so you get the
warning that is basically saying that that wal is from a non clean shutdown.
{noformat}
2015-08-18 00:26:15,093 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:26:15,095 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Log directory not found: File
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs
does not exist.
2015-08-18 00:26:15,107 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 1
- MASTER RESTARTED
2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Unable to read tracker for
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
- Missing trailer: size=9 startPos=9
2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 2
2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Lease acquired for flushLogId: 2
2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: No active entry found in state log
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
removing it
{noformat}
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
> 2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
> 2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: Recovering lease on dfs file
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> 2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: recoverLease=true, attempt=0 on
> file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> after 10ms
> 2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Opening state-log:
> FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
> isDirectory=false; length=9; replication=3; blocksize=134217728;
> modification_time=1439858084806; access_time=1439857575101; owner=hbase;
> group=hdfs; permission=rw-r--r--; isSymlink=false}
> 2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Unable to read tracker for
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> - Missing trailer: size=9 startPos=9
> 2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Roll new state log: 2
> 2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Lease acquired for flushLogId: 2
> 2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
> 2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: No active entry found in state log
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
> remov
[jira] [Comment Edited] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701650#comment-14701650
]
Matteo Bertozzi edited comment on HBASE-14240 at 8/18/15 5:33 PM:
--
is the master shutdown clean? i'm not able to understand it from the logs.
in theory on a clean master shutdown the trailer should be written to the wal
and the file closed. which seems not to be happened here. so you get the
warning that is basically saying that that wal is from a non clean shutdown.
{noformat}
- MASTER RESTARTED (first run with 1.1)
2015-08-18 00:26:15,093 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:26:15,095 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Log directory not found: File
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs
does not exist.
2015-08-18 00:26:15,107 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 1
- MASTER RESTARTED
2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Unable to read tracker for
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
- Missing trailer: size=9 startPos=9
2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 2
2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Lease acquired for flushLogId: 2
2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: No active entry found in state log
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
removing it
{noformat}
was (Author: mbertozzi):
is the master shutdown clean? i'm not able to understand it from the logs.
in theory on a clean master shutdown the trailer should be written to the wal
and the file closed. which seems not to be happened here. so you get the
warning that is basically saying that that wal is from a non clean shutdown.
{noformat}
2015-08-18 00:26:15,093 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:26:15,095 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Log directory not found: File
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs
does not exist.
2015-08-18 00:26:15,107 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 1
- MASTER RESTARTED
2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Unable to read tracker for
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
- Missing trailer: size=9 startPos=9
2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Roll new state log: 2
2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.WALProcedureStore: Lease acquired for flushLogId: 2
2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
wal.ProcedureWALFormatReader: No active entry found in state log
hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
removing it
{noformat}
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
>
[jira] [Updated] (HBASE-13376) Improvements to Stochastic load balancer
[ https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13376: --- Status: Patch Available (was: Reopened) > Improvements to Stochastic load balancer > > > Key: HBASE-13376 > URL: https://issues.apache.org/jira/browse/HBASE-13376 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 0.98.12, 1.0.0 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: 13376-v2.txt, 13376-v5.patch, 13376_4.patch, > HBASE-13376.patch, HBASE-13376_0.98.txt, HBASE-13376_0.98_v2.patch, > HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, > HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_3.patch, > HBASE-13376_3.patch, HBASE-13376_4.patch, HBASE-13376_5_branch-1.patch, > HBASE-13376_6_branch-1.patch, HBASE-13376_98.patch, > HBASE-13376_branch-1.patch, HBASE-13376_v3_0.98.patch > > > There are two things this jira tries to address: > 1. The locality picker in the stochastic balancer does not pick regions with > least locality as candidates for swap/move. So when any user configures > locality cost in the configs, the balancer does not always seems to move > regions with bad locality. > 2. When a cluster has equal number of loaded regions, it always picks the > first one. It should pick a random region on one of the equally loaded > servers. This improves a chance of finding a good candidate, when load picker > is invoked several times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-13376) Improvements to Stochastic load balancer
[ https://issues.apache.org/jira/browse/HBASE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-13376: > Improvements to Stochastic load balancer > > > Key: HBASE-13376 > URL: https://issues.apache.org/jira/browse/HBASE-13376 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.0.0, 0.98.12 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: 13376-v2.txt, 13376-v5.patch, 13376_4.patch, > HBASE-13376.patch, HBASE-13376_0.98.txt, HBASE-13376_0.98_v2.patch, > HBASE-13376_0.txt, HBASE-13376_1.txt, HBASE-13376_1_1.txt, > HBASE-13376_2.patch, HBASE-13376_2_branch-1.patch, HBASE-13376_3.patch, > HBASE-13376_3.patch, HBASE-13376_4.patch, HBASE-13376_5_branch-1.patch, > HBASE-13376_6_branch-1.patch, HBASE-13376_98.patch, > HBASE-13376_branch-1.patch, HBASE-13376_v3_0.98.patch > > > There are two things this jira tries to address: > 1. The locality picker in the stochastic balancer does not pick regions with > least locality as candidates for swap/move. So when any user configures > locality cost in the configs, the balancer does not always seems to move > regions with bad locality. > 2. When a cluster has equal number of loaded regions, it always picks the > first one. It should pick a random region on one of the equally loaded > servers. This improves a chance of finding a good candidate, when load picker > is invoked several times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HBASE-14239: - Assignee: Elliott Clark > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701689#comment-14701689
]
Matteo Bertozzi commented on HBASE-14240:
-
yeah, it doesn't look a clean shutdown. I don't see the log info saying
"Stopping the WAL Procedure Store" or any of the "Stopping" message we should
print on shutdown, and from the logs looks like the master is still doing work
when we get back to the hbase-daemon.sh script that says "Terminating master"
{noformat}
2015-08-18 00:34:44,688 WARN
[MASTER_META_SERVER_OPERATIONS-ndimiduk-rc-dist-1:6-1]
master.AssignmentManager: Can't move 1588230740, there is no destination server
available.
2015-08-18 00:34:44,688 WARN
[MASTER_META_SERVER_OPERATIONS-ndimiduk-rc-dist-1:6-1]
master.AssignmentManager: Unable to determine a plan to assign {ENCODED =>
1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
Tue Aug 18 00:34:44 UTC 2015 Terminating master
{noformat}
so, i'd say that the warn is ok. we can probably improve it by saying that the
missing trailer may be caused by a non clean shutdown or something like that
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
> 2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
> 2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: Recovering lease on dfs file
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> 2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: recoverLease=true, attempt=0 on
> file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> after 10ms
> 2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Opening state-log:
> FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
> isDirectory=false; length=9; replication=3; blocksize=134217728;
> modification_time=1439858084806; access_time=1439857575101; owner=hbase;
> group=hdfs; permission=rw-r--r--; isSymlink=false}
> 2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Unable to read tracker for
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> - Missing trailer: size=9 startPos=9
> 2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Roll new state log: 2
> 2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Lease acquired for flushLogId: 2
> 2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
> 2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: No active entry found in state log
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
> removing it
> 2015-08-18 00:42:59,711 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Remove log:
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> {noformat}
> These log lines are raised immediately after performing an in-place master
> upgrade from 0.98.0. Master initialized successfully and with no further
> errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14239: -- Attachment: HBASE-14239.patch > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14239: -- Affects Version/s: 1.2.0 Status: Patch Available (was: Open) All region moves were being treated as meta since that got tagged as hi-pri. > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14233) Branch-1.2 Merging regions can result in messed up tables
[
https://issues.apache.org/jira/browse/HBASE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701704#comment-14701704
]
Andrew Purtell commented on HBASE-14233:
Here is a list of commits found in branch-1.2 not in branch-1.1:
c728160 HBASE-14234 Exception encountered in WALProcedureStore#rollWriter()
should be properly handled
1588395 HBASE-14203 remove duplicate code getTableDescriptor in HTable (Heng
Chen)
05328c7 HBASE-14122 Client API for determining if server side supports cell
level security - ADDENDUM for javadoc fix
13d7250 HBASE-14166 Per-Region metrics can be stale -- ADDENDUM
fb91012 HBASE-14166 Per-Region metrics can be stale
adcb905 HBASE-10844 Coprocessor failure during batchmutation leaves the
memstore datastructs in an inconsistent state
34b706a HBASE-14098 Allow dropping caches behind compactions
2221021 HBASE-13985 Add configuration to skip validating HFile format when bulk
loading (Victor Xu)
2a5b5c7 HBASE-14122 Client API for determining if server side supports cell
level security
25bf721 HBASE-14201 hbck should not take a lock unless fixing errors
323e48a HBASE-14194 Undeprecate methods in ThriftServerRunner.HBaseHandler
a4c08f6 HBASE-13706 CoprocessorClassLoader should not exempt Hive classes
92c7bbf HBASE-14154 DFS Replication should be configurable at column family
level
707fba5 HBASE-14153 Typo in ProcedureManagerHost.MASTER_PROCEUDRE_CONF_KEY
(Konstantin Shvachko)
f0042a4 HBASE-14115 Fix resource leak in HMasterCommandLine (Yuhao Bi)
5bb840c HBASE-13971 Flushes stuck since 6 hours on a regionserver
de862f3 HBASE-14041 Do not clear MetaCache if a ThrottlingException is thrown
37e273b HBASE-14027 clean up multiple netty jars.
09a422e HBASE-14066 clean out old docbook docs from branch-1.
d360200 HBASE-13415 Procedure v2 - Use nonces for double submits from client
(Stephen Yuan Jiang)
928647b HBASE-14052 Mark a few methods in CellUtil audience private since only
make sense internally to hbase
bde52a9c HBASE-14053 Disable DLR in branch-1+
3e2636b HBASE-13927 Allow hbase-daemon.sh to conditionally redirect the log or
not
2bc5587 HBASE-13849 Remove restore and clone snapshot from the WebUI
042f53b HBASE-13646 HRegion#execService should not try to build incomplete
messages
8660a60 HBASE-14012 Double Assignment and Dataloss when ServerCrashProcedure
runs during Master failover
bf4d4ef HBASE-14015 Allow setting a richer state value when toString a pv2 --
ADDENDUM on branch-1 and derivatives
61b2694 HBASE-14015 Allow setting a richer state value when toString a pv2
ce5f25c HBASE-13978: Variable never assigned in
SimpleTotalOrderPartitioner.getPartition()
eeb864a HBASE-13990 make maven site generation work with jdk8
27c48d0 HBASE-13967 use jdk-specific profiles for jdk.tools version.
f2df4b7 HBASE-13963 Do not leak jdk.tools dependency from hbase-annotations
d476b56 HBASE-13947 Use MasterServices instead of Server in AssignmentManager
bce2eeb HBASE-13950 Add a NoopProcedureStore for testing
c680e2f HBASE-13964 Skip region normalization for tables under namespace quota
0e3fa8a HBASE-13920 Exclude org.apache.hadoop.hbase.protobuf.generated from
javadoc generation
911c0fb HBASE-13906 Improve handling of NeedUnmanagedConnectionException
a5240d2 HBASE-13898 AMMENDMENT Correct Javadoc errors in site
83f6474 HBASE-13898 correct additional javadoc failures under java 8
5d1603f HBASE-13103 [ergonomics] add region size balancing as a feature of
master
ef02245 HBASE-13917 Remove string comparison to identify request priority
67c229b HBASE-13918 Fix hbase:namespace description
> Branch-1.2 Merging regions can result in messed up tables
> -
>
> Key: HBASE-14233
> URL: https://issues.apache.org/jira/browse/HBASE-14233
> Project: HBase
> Issue Type: Sub-task
> Components: Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Priority: Critical
> Fix For: 1.2.0
>
>
> {code}
> 15/08/17 14:10:50 INFO master.RegionStates: Transition null to
> {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302}
> 15/08/17 14:10:57 INFO master.AssignmentManager: Failed to record merged
> region 6486b9b9409b25f10eb806ec3bad442d
> 15/08/17 14:10:57 ERROR master.AssignmentManager: Failed to transtion region
> from {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW, ts=1439845850422,
> server=hbase508.ash1.facebook.com,16020,1439844470302} to MERGE_PONR by
> hbase508.ash1.facebook.com,16020,1439844470302: Failed to record the merging
> in meta
> 15/08/17 14:11:08 WARN master.RegionStates: THIS SHOULD NOT HAPPEN:
> unexpected {6486b9b9409b25f10eb806ec3bad442d state=MERGING_NEW,
> ts=1439845857580, server=hbase508.ash1.facebook.com,16020,1439844470302}
> {code}
[jira] [Commented] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701712#comment-14701712
]
Nick Dimiduk commented on HBASE-14240:
--
Shutdown should have been clean; from my run log I used hbase-daemon.sh. It was
running 0.98.0, before, so there would have been no WAL Procedure Store to
stop. The installation was upgraded to 1.1.2RC0 and process started again.
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
> 2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
> 2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: Recovering lease on dfs file
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> 2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: recoverLease=true, attempt=0 on
> file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> after 10ms
> 2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Opening state-log:
> FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
> isDirectory=false; length=9; replication=3; blocksize=134217728;
> modification_time=1439858084806; access_time=1439857575101; owner=hbase;
> group=hdfs; permission=rw-r--r--; isSymlink=false}
> 2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Unable to read tracker for
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> - Missing trailer: size=9 startPos=9
> 2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Roll new state log: 2
> 2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Lease acquired for flushLogId: 2
> 2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
> 2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: No active entry found in state log
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
> removing it
> 2015-08-18 00:42:59,711 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Remove log:
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> {noformat}
> These log lines are raised immediately after performing an in-place master
> upgrade from 0.98.0. Master initialized successfully and with no further
> errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701718#comment-14701718 ] stack commented on HBASE-14239: --- Does all of HBASE-13351 "Annotate internal MasterRpcServices methods with admin priority" get undone by this change? If so, revert it? > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701719#comment-14701719 ] Elliott Clark commented on HBASE-14239: --- Testing on the cluster right now, I'll commit if it makes things better. > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701721#comment-14701721
]
Matteo Bertozzi commented on HBASE-14240:
-
the warn is not from a 98 to 1.1 restart. but from a 1.1 to a 1.1 restart.
if you look at the first piece of log i posted you see that at 00:26:15,093
you have the first 0.98 to 1.1 clean start.
then at 00:34:44 that master is not cleanly shutdown and at 00:42:59 you have
the second 1.1 startup that has the warn about the wal not closed properly
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
> 2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
> 2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: Recovering lease on dfs file
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> 2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: recoverLease=true, attempt=0 on
> file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> after 10ms
> 2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Opening state-log:
> FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
> isDirectory=false; length=9; replication=3; blocksize=134217728;
> modification_time=1439858084806; access_time=1439857575101; owner=hbase;
> group=hdfs; permission=rw-r--r--; isSymlink=false}
> 2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Unable to read tracker for
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> - Missing trailer: size=9 startPos=9
> 2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Roll new state log: 2
> 2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Lease acquired for flushLogId: 2
> 2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
> 2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: No active entry found in state log
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
> removing it
> 2015-08-18 00:42:59,711 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Remove log:
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> {noformat}
> These log lines are raised immediately after performing an in-place master
> upgrade from 0.98.0. Master initialized successfully and with no further
> errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701714#comment-14701714 ] Andrew Purtell commented on HBASE-14239: lgtm, Yeah, AnnotationReadingPriorityFunction has special case support for figuring out if ReportRegionStateTransition involves a system table that was being overridden by that annotation. > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[
https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701732#comment-14701732
]
Andrew Purtell commented on HBASE-14239:
bq. Does all of HBASE-13351 "Annotate internal MasterRpcServices methods with
admin priority" get undone by this change?
No, only one hunk out of many over 15 files, specifically:
{code}
@@ -1282,6 +1287,7 @@ public class MasterRpcServices extends RSRpcServices
}
@Override
+ @QosPriority(priority=HConstants.ADMIN_QOS)
public ReportRegionStateTransitionResponse
reportRegionStateTransition(RpcController c,
ReportRegionStateTransitionRequest req) throws ServiceException {
try {
{code}
> Branch-1.2 AM can get stuck when meta moves
> ---
>
> Key: HBASE-14239
> URL: https://issues.apache.org/jira/browse/HBASE-14239
> Project: HBase
> Issue Type: Sub-task
> Components: Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 1.2.0
>
> Attachments: HBASE-14239.patch
>
>
> When regions are moving master can get totally stuck trying to talk to meta.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14240) Unactionable WARN messages in wal.ProcedureWALFormatReader
[
https://issues.apache.org/jira/browse/HBASE-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701731#comment-14701731
]
Nick Dimiduk commented on HBASE-14240:
--
Ah yes, I see. There's nothing in the .out files to indicate abrupt shutdown.
> Unactionable WARN messages in wal.ProcedureWALFormatReader
> --
>
> Key: HBASE-14240
> URL: https://issues.apache.org/jira/browse/HBASE-14240
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Minor
> Attachments: ndimiduk-rc-dist-1.tgz
>
>
> While testing 1.1.2RC0, I notice the following messages in my master log.
> {noformat}
> 2015-08-18 00:42:59,646 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> procedure2.ProcedureExecutor: Starting procedure executor threads=5
> 2015-08-18 00:42:59,647 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
> 2015-08-18 00:42:59,652 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: Recovering lease on dfs file
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> 2015-08-18 00:42:59,664 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> util.FSHDFSUtils: recoverLease=true, attempt=0 on
> file=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> after 10ms
> 2015-08-18 00:42:59,665 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Opening state-log:
> FileStatus{path=hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log;
> isDirectory=false; length=9; replication=3; blocksize=134217728;
> modification_time=1439858084806; access_time=1439857575101; owner=hbase;
> group=hdfs; permission=rw-r--r--; isSymlink=false}
> 2015-08-18 00:42:59,676 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Unable to read tracker for
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> - Missing trailer: size=9 startPos=9
> 2015-08-18 00:42:59,703 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Roll new state log: 2
> 2015-08-18 00:42:59,705 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Lease acquired for flushLogId: 2
> 2015-08-18 00:42:59,711 WARN [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: nothing left to decode. exiting with missing EOF
> 2015-08-18 00:42:59,711 INFO [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.ProcedureWALFormatReader: No active entry found in state log
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log.
> removing it
> 2015-08-18 00:42:59,711 DEBUG [ndimiduk-rc-dist-1:6.activeMasterManager]
> wal.WALProcedureStore: Remove log:
> hdfs://ndimiduk-rc-dist-1.openstacklocal:8020/apps/hbase/data/MasterProcWALs/state-0001.log
> {noformat}
> These log lines are raised immediately after performing an in-place master
> upgrade from 0.98.0. Master initialized successfully and with no further
> errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[
https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701732#comment-14701732
]
Andrew Purtell edited comment on HBASE-14239 at 8/18/15 6:25 PM:
-
bq. Does all of HBASE-13351 "Annotate internal MasterRpcServices methods with
admin priority" get undone by this change?
No, only one hunk out of many over 15 files, specifically:
{code}
@@ -1282,6 +1287,7 @@ public class MasterRpcServices extends RSRpcServices
}
@Override
+ @QosPriority(priority=HConstants.ADMIN_QOS)
public ReportRegionStateTransitionResponse
reportRegionStateTransition(RpcController c,
ReportRegionStateTransitionRequest req) throws ServiceException {
try {
{code}
The global change needs tweaking but I'm not sure we have cause to undo it all
at this point.
was (Author: apurtell):
bq. Does all of HBASE-13351 "Annotate internal MasterRpcServices methods with
admin priority" get undone by this change?
No, only one hunk out of many over 15 files, specifically:
{code}
@@ -1282,6 +1287,7 @@ public class MasterRpcServices extends RSRpcServices
}
@Override
+ @QosPriority(priority=HConstants.ADMIN_QOS)
public ReportRegionStateTransitionResponse
reportRegionStateTransition(RpcController c,
ReportRegionStateTransitionRequest req) throws ServiceException {
try {
{code}
> Branch-1.2 AM can get stuck when meta moves
> ---
>
> Key: HBASE-14239
> URL: https://issues.apache.org/jira/browse/HBASE-14239
> Project: HBase
> Issue Type: Sub-task
> Components: Region Assignment
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 1.2.0
>
> Attachments: HBASE-14239.patch
>
>
> When regions are moving master can get totally stuck trying to talk to meta.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701739#comment-14701739 ] Elliott Clark commented on HBASE-14239: --- Report region in transition need to only be hi-pri if it's talking about meta. However we should check to make sure that other things that got annotated don't need a special case like reporting region in transition. I'll do that while I wait for itbll. > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14234) Exception encountered in WALProcedureStore#rollWriter() should be properly handled
[
https://issues.apache.org/jira/browse/HBASE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701745#comment-14701745
]
Hudson commented on HBASE-14234:
SUCCESS: Integrated in HBase-1.2-IT #96 (See
[https://builds.apache.org/job/HBase-1.2-IT/96/])
HBASE-14234 Exception encountered in WALProcedureStore#rollWriter() should be
properly handled (tedyu: rev c72816033a0a6ca4c37a589249160e246f916fae)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
> Exception encountered in WALProcedureStore#rollWriter() should be properly
> handled
> --
>
> Key: HBASE-14234
> URL: https://issues.apache.org/jira/browse/HBASE-14234
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14234-v1.txt
>
>
> Observed the following in recent Jenkins build
> (https://builds.apache.org/job/HBase-TRUNK/6732/console):
> {code}
> testWALfencingWithoutWALRolling(org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures)
> Time elapsed: 9.938 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: failed to create file
> /user/jenkins/test-data/0d9e3047-6bb1-4219-9ed2-5b9884176321/MasterProcWALs/state-0002.log
> for DFSClient_NONMAPREDUCE_-966558185_1 for client 127.0.0.1 because current
> leaseholder is trying to recreate file.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2589)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2386)
> {code}
> When file creation fails (e.g. due to RemoteException), we should handle the
> exception by returning false.
> Similar handling can be applied to failure in writing header.
> Thanks to [~mbertozzi] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14228) Close BufferedMutator and connection in MultiTableOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-14228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-14228: - Fix Version/s: 1.1.3 1.0.3 1.3.0 Adding a couple more fixVersions for branches where this should also apply. > Close BufferedMutator and connection in MultiTableOutputFormat > -- > > Key: HBASE-14228 > URL: https://issues.apache.org/jira/browse/HBASE-14228 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 1.1.1 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 > > Attachments: HBASE-14228-branch-1.patch > > > Close BufferedMutator and connection in MultiTableRecordWriter of > MultiTableOutputFormat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14239: -- Attachment: HBASE-14239-v1.patch Here's a few more methods. > Branch-1.2 AM can get stuck when meta moves > --- > > Key: HBASE-14239 > URL: https://issues.apache.org/jira/browse/HBASE-14239 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment >Affects Versions: 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 1.2.0 > > Attachments: HBASE-14239-v1.patch, HBASE-14239.patch > > > When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14228) Close BufferedMutator and connection in MultiTableOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-14228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701793#comment-14701793 ] Nick Dimiduk commented on HBASE-14228: -- Yes, this looks correct to me. +1 > Close BufferedMutator and connection in MultiTableOutputFormat > -- > > Key: HBASE-14228 > URL: https://issues.apache.org/jira/browse/HBASE-14228 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 1.1.1 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 > > Attachments: HBASE-14228-branch-1.patch > > > Close BufferedMutator and connection in MultiTableRecordWriter of > MultiTableOutputFormat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14234) Exception encountered in WALProcedureStore#rollWriter() should be properly handled
[
https://issues.apache.org/jira/browse/HBASE-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701797#comment-14701797
]
Hudson commented on HBASE-14234:
FAILURE: Integrated in HBase-TRUNK #6734 (See
[https://builds.apache.org/job/HBase-TRUNK/6734/])
HBASE-14234 Exception encountered in WALProcedureStore#rollWriter() should be
properly handled (tedyu: rev 71d3d24d8b5892121ac9c81606f3c71392a43e0b)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
> Exception encountered in WALProcedureStore#rollWriter() should be properly
> handled
> --
>
> Key: HBASE-14234
> URL: https://issues.apache.org/jira/browse/HBASE-14234
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14234-v1.txt
>
>
> Observed the following in recent Jenkins build
> (https://builds.apache.org/job/HBase-TRUNK/6732/console):
> {code}
> testWALfencingWithoutWALRolling(org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures)
> Time elapsed: 9.938 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: failed to create file
> /user/jenkins/test-data/0d9e3047-6bb1-4219-9ed2-5b9884176321/MasterProcWALs/state-0002.log
> for DFSClient_NONMAPREDUCE_-966558185_1 for client 127.0.0.1 because current
> leaseholder is trying to recreate file.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2589)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2386)
> {code}
> When file creation fails (e.g. due to RemoteException), we should handle the
> exception by returning false.
> Similar handling can be applied to failure in writing header.
> Thanks to [~mbertozzi] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
