[jira] [Commented] (HBASE-20503) [AsyncFSWAL] Failed to get sync result after 300000 ms for txid=160912, WAL system stuck?
[
https://issues.apache.org/jira/browse/HBASE-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460607#comment-16460607
]
stack commented on HBASE-20503:
---
.0001 is a test that reproduces the log pasted in above only this time there is
no hang-up of the WAL sub-system it seems to keep going. A file on which we
are currently recovering the lease is "archived" (by the periodic roll waller
thread interceding; it thinks any file whose edits have been synced is up for
rolling) which makes it so the NN does not recover the WAL lease; it fails with
FileNotFoundException (Default hadoop2, not hadoop3). The log roller should
wait until lease is recovered at least I'd think.
Let me try and find the original log to see if there are other clues as to what
about this combination locked up the WAL subsystem such that meta could no
longer take edits stuck on appending WAL.
> [AsyncFSWAL] Failed to get sync result after 30 ms for txid=160912, WAL
> system stuck?
> -
>
> Key: HBASE-20503
> URL: https://issues.apache.org/jira/browse/HBASE-20503
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments:
> 0001-HBASE-20503-AsyncFSWAL-Failed-to-get-sync-result-aft.patch
>
>
> Scale test. Startup w/ 30k regions over ~250nodes. This RS is trying to
> furiously open regions assigned by Master. It is importantly carrying
> hbase:meta. Twenty minutes in, meta goes dead after an exception up out
> AsyncFSWAL. Process had been restarted so I couldn't get a thread dump.
> Suspicious is we archive a WAL and we get a FNFE because we got to access WAL
> in old location. [~Apache9] mind taking a look? Does this FNFE rolling kill
> the WAL sub-system? Thanks.
> DFS complaining on file open for a few files getting blocks from remote dead
> DNs: e.g. {{2018-04-25 10:05:21,506 WARN
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
> remote block reader.
> java.net.ConnectException: Connection refused}}
> AsyncFSWAL complaining: "AbstractFSWAL: Slow sync cost: 103 ms" .
> About ten minutes in, we get this:
> {code}
> 2018-04-25 10:15:16,532 WARN
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL: sync failed
> java.io.IOException: stream already broken
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:424)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:513)
>
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:134)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:364)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:547)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-25 10:15:16,680 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Rolled WAL
> /hbase/WALs/vc0205.halxg.cloudera.com,22101,1524675808073/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524676253923.meta
> with entries=10819, filesize=7.57 MB; new WAL
> /hbase/WALs/vc0205.halxg.cloudera.com,22101,1524675808073/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524676516535.meta
> 2018-04-25 10:15:16,680 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Archiving
> hdfs://ns1/hbase/WALs/vc0205.halxg.cloudera.com,22101,1524675808073/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524675848653.meta
> to
> hdfs://ns1/hbase/oldWALs/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524675848653.meta
> 2018-04-25 10:15:16,686 WARN
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter: Failed to
> write trailer, non-fatal, continuing...
> java.io.IOException: stream already broken
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:424)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:513)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:210)
> at
> org.apache.ha
[jira] [Updated] (HBASE-20503) [AsyncFSWAL] Failed to get sync result after 300000 ms for txid=160912, WAL system stuck?
[
https://issues.apache.org/jira/browse/HBASE-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20503:
--
Attachment: 0001-HBASE-20503-AsyncFSWAL-Failed-to-get-sync-result-aft.patch
> [AsyncFSWAL] Failed to get sync result after 30 ms for txid=160912, WAL
> system stuck?
> -
>
> Key: HBASE-20503
> URL: https://issues.apache.org/jira/browse/HBASE-20503
> Project: HBase
> Issue Type: Bug
> Components: wal
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments:
> 0001-HBASE-20503-AsyncFSWAL-Failed-to-get-sync-result-aft.patch,
> 0001-HBASE-20503-AsyncFSWAL-Failed-to-get-sync-result-aft.patch
>
>
> Scale test. Startup w/ 30k regions over ~250nodes. This RS is trying to
> furiously open regions assigned by Master. It is importantly carrying
> hbase:meta. Twenty minutes in, meta goes dead after an exception up out
> AsyncFSWAL. Process had been restarted so I couldn't get a thread dump.
> Suspicious is we archive a WAL and we get a FNFE because we got to access WAL
> in old location. [~Apache9] mind taking a look? Does this FNFE rolling kill
> the WAL sub-system? Thanks.
> DFS complaining on file open for a few files getting blocks from remote dead
> DNs: e.g. {{2018-04-25 10:05:21,506 WARN
> org.apache.hadoop.hdfs.client.impl.BlockReaderFactory: I/O error constructing
> remote block reader.
> java.net.ConnectException: Connection refused}}
> AsyncFSWAL complaining: "AbstractFSWAL: Slow sync cost: 103 ms" .
> About ten minutes in, we get this:
> {code}
> 2018-04-25 10:15:16,532 WARN
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL: sync failed
> java.io.IOException: stream already broken
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:424)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:513)
>
>
>
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:134)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:364)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:547)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-04-25 10:15:16,680 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Rolled WAL
> /hbase/WALs/vc0205.halxg.cloudera.com,22101,1524675808073/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524676253923.meta
> with entries=10819, filesize=7.57 MB; new WAL
> /hbase/WALs/vc0205.halxg.cloudera.com,22101,1524675808073/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524676516535.meta
> 2018-04-25 10:15:16,680 INFO
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL: Archiving
> hdfs://ns1/hbase/WALs/vc0205.halxg.cloudera.com,22101,1524675808073/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524675848653.meta
> to
> hdfs://ns1/hbase/oldWALs/vc0205.halxg.cloudera.com%2C22101%2C1524675808073.meta.1524675848653.meta
> 2018-04-25 10:15:16,686 WARN
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter: Failed to
> write trailer, non-fatal, continuing...
> java.io.IOException: stream already broken
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:424)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:513)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:210)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:166)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:201)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:233)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:143)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$executeClose$8(AsyncFSWAL.java:742)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolE
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Attachment: HBASE-20169.master.001.ADDENDUM.patch
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460610#comment-16460610
]
stack commented on HBASE-20169:
---
Failures are probably related but retrying in case not.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Attachments: HBASE-20169.branch-2.001.patch,
> HBASE-20169.branch-2.002.patch, HBASE-20169.branch-2.003.patch,
> HBASE-20169.branch-2.004.patch, HBASE-20169.branch-2.005.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20426) Give up replicating anything in S state
[ https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20426: -- Attachment: HBASE-20426-HBASE-19064-v2.patch > Give up replicating anything in S state > --- > > Key: HBASE-20426 > URL: https://issues.apache.org/jira/browse/HBASE-20426 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v2.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-UT.patch > > > When we transit the remote S cluster to DA, and then transit the old A > cluster to S, it is possible that we still have some entries which have not > been replicated yet for the old A cluster, and then the async replication > will be blocked. > And this may also lead to data inconsistency after we transit it to DA back > later as these entries will be replicated again, but the new data which are > replicated from the remote cluster will not be replicated back, which > introduce a whole in the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20426) Give up replicating anything in S state
[
https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460662#comment-16460662
]
Hadoop QA commented on HBASE-20426:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-20426 does not apply to HBASE-19064. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20426 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921548/HBASE-20426-HBASE-19064-v2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12702/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Give up replicating anything in S state
> ---
>
> Key: HBASE-20426
> URL: https://issues.apache.org/jira/browse/HBASE-20426
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: HBASE-19064
>
> Attachments: HBASE-20426-HBASE-19064-v1.patch,
> HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch,
> HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch,
> HBASE-20426-HBASE-19064-v2.patch, HBASE-20426-HBASE-19064.patch,
> HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch,
> HBASE-20426-UT.patch
>
>
> When we transit the remote S cluster to DA, and then transit the old A
> cluster to S, it is possible that we still have some entries which have not
> been replicated yet for the old A cluster, and then the async replication
> will be blocked.
> And this may also lead to data inconsistency after we transit it to DA back
> later as these entries will be replicated again, but the new data which are
> replicated from the remote cluster will not be replicated back, which
> introduce a whole in the replication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20426) Give up replicating anything in S state
[ https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20426: -- Attachment: (was: HBASE-20426-HBASE-19064-v2.patch) > Give up replicating anything in S state > --- > > Key: HBASE-20426 > URL: https://issues.apache.org/jira/browse/HBASE-20426 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-UT.patch > > > When we transit the remote S cluster to DA, and then transit the old A > cluster to S, it is possible that we still have some entries which have not > been replicated yet for the old A cluster, and then the async replication > will be blocked. > And this may also lead to data inconsistency after we transit it to DA back > later as these entries will be replicated again, but the new data which are > replicated from the remote cluster will not be replicated back, which > introduce a whole in the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20426) Give up replicating anything in S state
[ https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20426: -- Attachment: HBASE-20426-HBASE-19064-v2.patch > Give up replicating anything in S state > --- > > Key: HBASE-20426 > URL: https://issues.apache.org/jira/browse/HBASE-20426 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v2.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-UT.patch > > > When we transit the remote S cluster to DA, and then transit the old A > cluster to S, it is possible that we still have some entries which have not > been replicated yet for the old A cluster, and then the async replication > will be blocked. > And this may also lead to data inconsistency after we transit it to DA back > later as these entries will be replicated again, but the new data which are > replicated from the remote cluster will not be replicated back, which > introduce a whole in the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
[ https://issues.apache.org/jira/browse/HBASE-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad Arshad reassigned HBASE-17115: --- Assignee: Mohammad Arshad > HMaster/HRegion Info Server does not honour admin.acl > - > > Key: HBASE-17115 > URL: https://issues.apache.org/jira/browse/HBASE-17115 > Project: HBase > Issue Type: Bug >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > > Currently there is no way to enable protected URLs like /jmx, /conf only > for admins. This is applicable for both Master and RegionServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Arshad reassigned HBASE-19423:
---
Assignee: Mohammad Arshad
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Labels: Replication, WAL
> Fix For: 1.4.0, 1.3.3
>
> Attachments: HBASE-19423-branch-1.3-001.patch,
> HBASE-19423-branch-1.4-001.patch, HBASE-19423-master-001-test.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20518) Need to serialize the enabled field for UpdatePeerConfigProcedure
Duo Zhang created HBASE-20518: - Summary: Need to serialize the enabled field for UpdatePeerConfigProcedure Key: HBASE-20518 URL: https://issues.apache.org/jira/browse/HBASE-20518 Project: HBase Issue Type: Sub-task Components: Replication Affects Versions: 3.0.0, 2.1.0 Reporter: Duo Zhang Fix For: 3.0.0, 2.1.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20426) Give up replicating anything in S state
[ https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20426: -- Attachment: HBASE-20426-HBASE-19064-v3.patch > Give up replicating anything in S state > --- > > Key: HBASE-20426 > URL: https://issues.apache.org/jira/browse/HBASE-20426 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v2.patch, HBASE-20426-HBASE-19064-v3.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-UT.patch > > > When we transit the remote S cluster to DA, and then transit the old A > cluster to S, it is possible that we still have some entries which have not > been replicated yet for the old A cluster, and then the async replication > will be blocked. > And this may also lead to data inconsistency after we transit it to DA back > later as these entries will be replicated again, but the new data which are > replicated from the remote cluster will not be replicated back, which > introduce a whole in the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20426) Give up replicating anything in S state
[
https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460788#comment-16460788
]
Hadoop QA commented on HBASE-20426:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} hbase-server: The patch generated 1 new + 4 unchanged -
1 fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 16s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
0m 59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 59s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20426 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921550/HBASE-20426-HBASE-19064-v2.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopch
[jira] [Updated] (HBASE-20476) Open sequence number could go backwards in AssignProcedure
[ https://issues.apache.org/jira/browse/HBASE-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20476: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2.0. Thanks [~stack] for reviewing. > Open sequence number could go backwards in AssignProcedure > -- > > Key: HBASE-20476 > URL: https://issues.apache.org/jira/browse/HBASE-20476 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.1.0, 2.0.0 >Reporter: Zheng Hu >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20476-addlog.patch, HBASE-20476-branch-2.0.patch, > HBASE-20476-v1.patch, HBASE-20476-v2.patch, HBASE-20476.patch > > > See > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460852#comment-16460852
]
Hadoop QA commented on HBASE-20169:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 22s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 35s{color}
| {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}156m 12s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}209m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.procedure2.TestStateMachineProcedure |
| | hadoop.hbase.procedure2.TestChildProcedures |
| | hadoop.hbase.procedure2.TestProcedureNonce |
| | hadoop.hbase.procedure2.TestProcedureRecovery |
| | hadoop.hbase.procedure2.TestProcedureEvents |
| | hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure |
| | hadoop.hbase.master.procedure.TestDeleteNamespaceProcedure |
| | hadoop.hbase.master.locking.TestLockProcedure |
| | hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.master.procedure.TestCreateNamespaceProcedure |
| | hadoop.hbase.master.procedure.TestModifyNamespaceProcedure |
| | hadoop.hbase.master.procedure.TestDisab
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460868#comment-16460868
]
Hudson commented on HBASE-20507:
Results for branch branch-2.0
[build #249 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20484) Remove the unnecessary autoboxing in FilterListBase
[
https://issues.apache.org/jira/browse/HBASE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460867#comment-16460867
]
Hudson commented on HBASE-20484:
Results for branch branch-2.0
[build #249 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove the unnecessary autoboxing in FilterListBase
> ---
>
> Key: HBASE-20484
> URL: https://issues.apache.org/jira/browse/HBASE-20484
> Project: HBase
> Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Guangxu Cheng
>Priority: Trivial
> Labels: beginner, beginners
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20484.master.001.patch
>
>
> {code:java}
> protected static boolean checkAndGetReversed(List rowFilters, boolean
> defaultValue) {
> if (rowFilters.isEmpty()) {
> return defaultValue;
> }
> Boolean retValue = rowFilters.get(0).isReversed(); // this one
> boolean allEqual =
> rowFilters.stream().map(Filter::isReversed).allMatch(retValue::equals);
> if (!allEqual) {
> throw new IllegalArgumentException("Filters in the list must have the
> same reversed flag");
> }
> return retValue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20327) When qualifier is not specified, append and incr operation do not work (shell)
[
https://issues.apache.org/jira/browse/HBASE-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460866#comment-16460866
]
Hudson commented on HBASE-20327:
Results for branch branch-2.0
[build #249 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/249//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> When qualifier is not specified, append and incr operation do not work (shell)
> --
>
> Key: HBASE-20327
> URL: https://issues.apache.org/jira/browse/HBASE-20327
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20327.master.001.patch,
> HBASE-20327.master.002.patch, HBASE-20327.master.003.patch,
> HBASE-20327.master.004.patch
>
>
> Running the example commands specified in shell docs for "append" and "incr"
> throw following error:
> {code:java}
> ERROR: Failed to provide both column family and column qualifier for
> append{code}
> {code:java}
> ERROR: Failed to provide both column family and column qualifier for
> incr{code}
> While running the same command via java does not require the user to provide
> both column and qualifier and works smoothly.
>
> Steps to reproduce:
> 1) APPEND
> {code:java}
> hbase(main):002:0> create 't1', 'c1', 'c2'
> Created table t1
> Took 0.8151 seconds
>
> hbase(main):003:0> append 't1', 'r1', 'c1', 'value'
> ERROR: Failed to provide both column family and column qualifier for append
> Appends a cell 'value' at specified table/row/column coordinates.
> hbase> append 't1', 'r1', 'c1', 'value', ATTRIBUTES=>{'mykey'=>'myvalue'}
> hbase> append 't1', 'r1', 'c1', 'value', {VISIBILITY=>'PRIVATE|SECRET'}
> The same commands also can be run on a table reference. Suppose you had a
> reference
> t to table 't1', the corresponding command would be:
> hbase> t.append 'r1', 'c1', 'value', ATTRIBUTES=>{'mykey'=>'myvalue'}
> hbase> t.append 'r1', 'c1', 'value', {VISIBILITY=>'PRIVATE|SECRET'}
> Took 0.0326 seconds
> hbase(main):004:0> scan 't1'
> ROW COLUMN+CELL
>
> 0 row(s)
> Took 0.1273 seconds
> {code}
> While the same command would run if we run the following java code:
> {code:java}
> try (Connection connection = ConnectionFactory.createConnection(config);
> Admin admin = connection.getAdmin();) {
> Table table = connection.getTable(TableName.valueOf("t1"));
> Append a = new Append(Bytes.toBytes("r1"));
> a.addColumn(Bytes.toBytes("c1"), null, Bytes.toBytes("value"));
> table.append(a);
> }{code}
> Scan result after executing java code:
> {code:java}
> hbase(main):005:0> scan 't1'
> ROW COLUMN+CELL
>
> r1 column=c1:, timestamp=1522649623090,
> value=value
> 1 row(s)
> Took 0.0188 seconds
> {code}
>
> 2) INCREMENT:
> Similarly in case of increment, we get the following error (shell):
> {code:java}
> hbase(main):006:0> incr 't1', 'r2', 'c1', 111
> ERROR: Failed to provide both column family and column qualifier for incr
> Increments a cell 'value' at specified table/row/column coordinates.
> To increment a cell value in table 'ns1:t1' or 't1' at row 'r1' under column
> 'c1' by 1 (can be omitted) or 10 do:
> hbase> incr 'ns1:t1', 'r1', 'c1'
> hbase> incr 't1', 'r1', 'c1'
> hbase> incr 't1', 'r1', 'c1', 1
> hbase> incr 't1', 'r1', 'c1', 10
> hbase> incr 't1', 'r1', 'c1', 10,
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460887#comment-16460887
]
Hudson commented on HBASE-20507:
Results for branch branch-2
[build #685 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/685//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20481) Add a setSerial method to ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20481: - Attachment: HBASE-20481.v1.patch > Add a setSerial method to ReplicationEndpoint > - > > Key: HBASE-20481 > URL: https://issues.apache.org/jira/browse/HBASE-20481 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20481.v1.patch > > > When debugging HBASE-20475, [~openinx] found that the > HBaseInterClusterReplicationEndpoint may send the entries for the same > regions concurrently, which breaks the serial replication. > As long as we can have multiple ReplicationEndpoint implementation, just fix > HBaseInterClusterReplicationEndpoint is not enough, we need to add a > setSerial method to ReplicationEndpoint, to tell the implementation that you > should keep the order of the entries from the same region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20481) Add a setSerial method to ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-20481: - Status: Patch Available (was: Open) > Add a setSerial method to ReplicationEndpoint > - > > Key: HBASE-20481 > URL: https://issues.apache.org/jira/browse/HBASE-20481 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20481.v1.patch > > > When debugging HBASE-20475, [~openinx] found that the > HBaseInterClusterReplicationEndpoint may send the entries for the same > regions concurrently, which breaks the serial replication. > As long as we can have multiple ReplicationEndpoint implementation, just fix > HBaseInterClusterReplicationEndpoint is not enough, we need to add a > setSerial method to ReplicationEndpoint, to tell the implementation that you > should keep the order of the entries from the same region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20506) Add doc and test for unused RetryCounter, useful-looking utility
[
https://issues.apache.org/jira/browse/HBASE-20506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460944#comment-16460944
]
Hudson commented on HBASE-20506:
Results for branch master
[build #318 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/318/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add doc and test for unused RetryCounter, useful-looking utility
>
>
> Key: HBASE-20506
> URL: https://issues.apache.org/jira/browse/HBASE-20506
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.1.0
>
> Attachments: 20506.txt, HBASE-20506.master.001.patch,
> HBASE-20506.master.002.patch
>
>
> I thought I could use RetryCounter, old facility added years ago, for doing
> backoff calculations. In the end, it didn't work for me because it is lacking
> pb serialization. While trying to use it, I added a bit of doc and a test.
> Might help the next dev that trips along this way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15317) document release announcement template
[
https://issues.apache.org/jira/browse/HBASE-15317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460945#comment-16460945
]
Hudson commented on HBASE-15317:
Results for branch master
[build #318 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/318/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> document release announcement template
> --
>
> Key: HBASE-15317
> URL: https://issues.apache.org/jira/browse/HBASE-15317
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-15317.0.patch, HBASE-15317.1.patch
>
>
> Our release docs should include a release announcement template for folks to
> use (and suggested email lists to send to)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20507) Do not need to call recoverLease on the broken file when we fail to create a wal writer
[
https://issues.apache.org/jira/browse/HBASE-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460946#comment-16460946
]
Hudson commented on HBASE-20507:
Results for branch master
[build #318 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/318/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/318//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Do not need to call recoverLease on the broken file when we fail to create a
> wal writer
> ---
>
> Key: HBASE-20507
> URL: https://issues.apache.org/jira/browse/HBASE-20507
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HBASE-20507.patch
>
>
> I tried locally with a UT, if we overwrite a file which is currently being
> written, the old file will be completed and then deleted. If you call close
> on the previous file, a no lease exception will be thrown which means that
> the file has already been completed.
> So we do not need to close a file if it will be overwritten immediately,
> since recoverLease may take a very long time...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20426) Give up replicating anything in S state
[
https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460965#comment-16460965
]
Hadoop QA commented on HBASE-20426:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
9s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
38s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 1 new + 4 unchanged
- 1 fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 52s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}150m
18s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20426 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921563/HBASE-20426-HBASE-19064-v3.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 418a6e494e20 3.13.0-137-generi
[jira] [Commented] (HBASE-20481) Add a setSerial method to ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460966#comment-16460966 ] Duo Zhang commented on HBASE-20481: --- Sorry, just notice that when calling ReplicationEndpoint.init, we pass a Context parameter, which contains a ReplicationPeer field, so the current API is enough to tell the ReplicationEndpoint whether the peer is serial or not... And for your solution, I think it is too strict. You make everything serial, but actually, we only need the serial guarantee inside a region, for different regions we can replicate them parallelly... > Add a setSerial method to ReplicationEndpoint > - > > Key: HBASE-20481 > URL: https://issues.apache.org/jira/browse/HBASE-20481 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20481.v1.patch > > > When debugging HBASE-20475, [~openinx] found that the > HBaseInterClusterReplicationEndpoint may send the entries for the same > regions concurrently, which breaks the serial replication. > As long as we can have multiple ReplicationEndpoint implementation, just fix > HBaseInterClusterReplicationEndpoint is not enough, we need to add a > setSerial method to ReplicationEndpoint, to tell the implementation that you > should keep the order of the entries from the same region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20359) why Comparator extends comparable?
[
https://issues.apache.org/jira/browse/HBASE-20359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460974#comment-16460974
]
maoling commented on HBASE-20359:
-
[~mdrob]
add below config in the hbase\hbase-build-configuration\pom.xml
{code:java}
-Xep:ComparableAndComparator:ERROR
{code}
maven build success.It seems that no codes violate the ComparableAndComparator
rule?
> why Comparator extends comparable?
> --
>
> Key: HBASE-20359
> URL: https://issues.apache.org/jira/browse/HBASE-20359
> Project: HBase
> Issue Type: Improvement
>Reporter: Nick.han
>Priority: Minor
>
> Comparable is an interface,object implements this interface can be compare
> with some other object, Comparator is a tool for compare object,the tool
> itself can be compare is useless, so why a tool implement the comparable
> interface?Do we have to improve this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20414) TestLockProcedure#testMultipleLocks may fail on slow machine
[
https://issues.apache.org/jira/browse/HBASE-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20414:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the review, Chia-ping and Ram.
> TestLockProcedure#testMultipleLocks may fail on slow machine
>
>
> Key: HBASE-20414
> URL: https://issues.apache.org/jira/browse/HBASE-20414
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 20414.v1.txt, 20414.v2.txt
>
>
> Here was recent failure :
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/172/testReport/junit/org.apache.hadoop.hbase.master.locking/TestLockProcedure/health_checks___yetus_jdk8_hadoop2_checks___testMultipleLocks/
> {code}
> java.lang.AssertionError: expected: but was:
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.sendHeartbeatAndCheckLocked(TestLockProcedure.java:221)
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.testMultipleLocks(TestLockProcedure.java:311)
> {code}
> In the test output, we can see this:
> {code}
> 2018-04-13 20:19:54,230 DEBUG [Time-limited test]
> locking.TestLockProcedure(225): Proc id 22 : LOCKED.
> ...
> 2018-04-13 20:19:55,529 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(865): Stored pid=26, state=RUNNABLE;
> org.apache.hadoop.hbase.master.locking.LockProcedure
> regions=a7f9adfd047350eabb199a39564ba4db,c1e297609590b707233a2f9c8bb51fa1,
> type=EXCLUSIVE
> 2018-04-13 20:19:56,230 DEBUG [ProcExecTimeout] locking.LockProcedure(207):
> Timeout failure ProcedureEvent for pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE, ready=false, [pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE]
> {code}
> After the pid=26 log, the code does this (1 second wait):
> {code}
> // Assert tables & region locks are waiting because of namespace lock.
> Thread.sleep(HEARTBEAT_TIMEOUT / 2);
> {code}
> On a slow machine (in the case above), there was only 730 msec between the
> queueing of regionsLock2 and the WAITING_TIMEOUT state of the nsLock. The 1
> second wait was too long, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20481) Add a setSerial method to ReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1646#comment-1646
]
Hadoop QA commented on HBASE-20481:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
59s{color} | {color:red} hbase-server: The patch generated 2 new + 16 unchanged
- 1 fixed = 18 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 59s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m
15s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921574/HBASE-20481.v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d7c6a1f59fd1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5bdb52af2f |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/12705/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12705/testReport/ |
| Max. process+thread count | 4644 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://
[jira] [Commented] (HBASE-20476) Open sequence number could go backwards in AssignProcedure
[
https://issues.apache.org/jira/browse/HBASE-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461222#comment-16461222
]
Hudson commented on HBASE-20476:
Results for branch branch-2.0
[build #250 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/250/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/250//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/250//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/250//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Open sequence number could go backwards in AssignProcedure
> --
>
> Key: HBASE-20476
> URL: https://issues.apache.org/jira/browse/HBASE-20476
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
>Reporter: Zheng Hu
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20476-addlog.patch, HBASE-20476-branch-2.0.patch,
> HBASE-20476-v1.patch, HBASE-20476-v2.patch, HBASE-20476.patch
>
>
> See
> https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20519) [Chaos] Add more chaos options
stack created HBASE-20519: - Summary: [Chaos] Add more chaos options Key: HBASE-20519 URL: https://issues.apache.org/jira/browse/HBASE-20519 Project: HBase Issue Type: Umbrella Components: integration tests Reporter: stack Our Chaos menu is "drawing room polite" given the variety of failures available out in the wild world of deploys. Other possible items to add (could do as subtasks of this umbrella) taken from a recent [interesting read on how TiDB does its chaos|https://thenewstack.io/chaos-tools-and-techniques-for-testing-the-tidb-distributed-newsql-database/]: * Send SIGSTOP to hang or SIGCONT to resume the process. Use `renice` to adjust the process priority or use `setpriority` for the threads of the process. * Max out the CPU. * Use `iptables` or `tc` to drop or reject the network packages or delay the network packages. * Use `tc` to reorder the network packages and use a proxy to reorder the gRPC requests. * Use `iperf` to take all network throughput. * Use `libfuse` to mount a file system and do the I/O fault injection. * Link `libfiu` to do the I/O fault injection. * Use `rm -rf` forcbily to remove all data. * Use `echo 0 > file` to damage a file. * Copy a huge file to create the `NoSpace` problem. The article includes other interesting possibilities: exploiting the kernels fault injection mechanism or scripting systemtap to mess with nodes. It also describes how they automate their chaos-making. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Attachment: HBASE-20169.branch-2.0.001.patch
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Attachments: HBASE-20169.branch-2.0.001.patch,
> HBASE-20169.branch-2.001.patch, HBASE-20169.branch-2.002.patch,
> HBASE-20169.branch-2.003.patch, HBASE-20169.branch-2.004.patch,
> HBASE-20169.branch-2.005.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461354#comment-16461354
]
stack commented on HBASE-20169:
---
HBASE-20169.branch-2.0.001.patch patch is new version of ADDENDUM. It fixes
the test. The PE testing utility wants to be able to overwrite thread group and
timer thread in PE so it can do its dirty tricks testing.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
> Attachments: HBASE-20169.branch-2.0.001.patch,
> HBASE-20169.branch-2.001.patch, HBASE-20169.branch-2.002.patch,
> HBASE-20169.branch-2.003.patch, HBASE-20169.branch-2.004.patch,
> HBASE-20169.branch-2.005.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
stack created HBASE-20520: - Summary: Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize Key: HBASE-20520 URL: https://issues.apache.org/jira/browse/HBASE-20520 Project: HBase Issue Type: Sub-task Components: conf Reporter: stack Good one found by our [~mdrob]. Problem in the parent issue where failed attempt doubling default block size but halving the size at which roll -- so we end up in roughly same place only we make less small WALs in those cases where we are writing furiously and fail to roll before starting a new block. >From Drob: ".../we drop default log roll to 0.5, but we don't actually >increase the block size... AbstractProtobufLogWriter.java:164 still uses >default HDFS block sizing and AbstractFSWAL.java:408 block size is only used >in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461416#comment-16461416 ] Mike Drob commented on HBASE-20520: --- You think the too-early log roll maybe had perf impact on our write loads? > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Priority: Major > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461427#comment-16461427 ] ramkrishna.s.vasudevan commented on HBASE-20520: In the logs I noticed that WAL A gets renamed to WAL B and in the immediate second WAL B gets renamed to WAL C. Is that because of the issue mentioned here? > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Priority: Major > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461468#comment-16461468 ] stack commented on HBASE-20520: --- bq. You think the too-early log roll maybe had perf impact on our write loads? There's a hiccup each time we roll where writes are blocked for a short while so yes, this will have had some impact but small in the scheme of things I'd say given the new WAL is made ahead of the roll and dealing with the old is backgrounded but let me remeasure (There is also our blocking when too many WALs but in my perf tests I'd turned that off). bq. In the logs I noticed that WAL A gets renamed to WAL B and in the immediate second WAL B gets renamed to WAL C. Is that because of the issue mentioned here? Example please [~ram_krish] (HBASE-20503 ? Where Archiving seems to cut in prematurely?) > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Priority: Major > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20520: -- Attachment: HBASE-20520.branch-2.0.001.patch > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Priority: Major > Attachments: HBASE-20520.branch-2.0.001.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20520: -- Assignee: stack Fix Version/s: 2.0.1 Status: Patch Available (was: Open) > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20514) On Master restart if table is stuck in DISABLING state, CLOSED regions should not be considered stuck in-transition
[ https://issues.apache.org/jira/browse/HBASE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20514: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.0+. Checkstyle complaint is about empty statement inside clause which is how it was before. I tried adding an empty statement but checkstyle didn't like that either. > On Master restart if table is stuck in DISABLING state, CLOSED regions should > not be considered stuck in-transition > --- > > Key: HBASE-20514 > URL: https://issues.apache.org/jira/browse/HBASE-20514 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Fix For: 2.0.1 > > Attachments: hbase-20514.master.001.patch > > > When master is restarted, in AssignmentManager#loadMeta(), if table is in > DISABLED state nothing is done for regions in CLOSED state. But if table is > stuck in DISABLING state then CLOSED regions are considered as stuck > in-transition. CLOSED regions of DISABLING/ DISABLED table can be handled the > same way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20518) Need to serialize the enabled field for UpdatePeerConfigProcedure
[ https://issues.apache.org/jira/browse/HBASE-20518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461524#comment-16461524 ] Nihal Jain commented on HBASE-20518: Hi [~Apache9], can I take this one? > Need to serialize the enabled field for UpdatePeerConfigProcedure > - > > Key: HBASE-20518 > URL: https://issues.apache.org/jira/browse/HBASE-20518 > Project: HBase > Issue Type: Sub-task > Components: Replication >Affects Versions: 3.0.0, 2.1.0 >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[
https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461560#comment-16461560
]
Hadoop QA commented on HBASE-20520:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
9s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
2s{color} | {color:green} branch-2.0 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
23s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
29s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 12m
45s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 45s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
51s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 2m
21s{color} | {color:red} patch has 19 errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m
28s{color} | {color:red} The patch causes 19 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m
54s{color} | {color:red} The patch causes 19 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m
32s{color} | {color:red} The patch causes 19 errors with Hadoop v3.0.0. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 58s{color}
| {color:red} root in the patch failed. {color} |
| {colo
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461577#comment-16461577
]
Hadoop QA commented on HBASE-20169:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 41s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}110m
28s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:369877d |
| JIRA Issue | HBASE-20169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921611/HBASE-20169.branch-2.0.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2c681d1c227d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.0 / 8886c844f9 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T
[jira] [Commented] (HBASE-20295) TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
[
https://issues.apache.org/jira/browse/HBASE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461602#comment-16461602
]
Mike Drob commented on HBASE-20295:
---
[~tedyu], [~menjin], [~elserj], [~stack] - Question about this implementation,
sorry for showing up a month late to the issue...
The patch adds:
{noformat}
Configuration hConf = context.getConfiguration();
if(hConf == null) {
hConf = this.conf;
}
{noformat}
Are we doing this backwards? If the concern is that sometimes getConf() returns
null, shouldn't we be using that first and checking it's output instead of
pulling the conf from the configuration?
I'm having a pig script that fails against hbase-2.0 with this patch included,
if I revert the patch then my pig works again.
> TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
> ---
>
> Key: HBASE-20295
> URL: https://issues.apache.org/jira/browse/HBASE-20295
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.4.0
> Environment: Spark 2.2.1, HBase 1.4.0
>Reporter: Michael Jin
>Assignee: Michael Jin
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20295.branch-1.4.001.patch,
> HBASE-20295.master.001.patch, HBASE-20295.master.002.patch,
> HBASE-20295.master.003.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I am using spark write data to HBase by using RDD.
> saveAsNewAPIHadoopDataset function, it works fine with hbase 1.3.1, but when
> update my hbase dependency to 1.4.0 in pom.xml, it throw
> java.lang.NullPointerException, it is caused by a logic error in
> TableOutputFormat.checkOutputSpecs function, please check below details:
> first let's take a look at SparkHadoopMapReduceWriter.write function in
> SparkHadoopMapReduceWriter.scala
> {code:java}
> // SparkHadoopMapReduceWriter.write
> (org.apache.spark.internal.io.SparkHadoopMapReduceWriter.scala)
> def write[K, V: ClassTag](
> rdd: RDD[(K, V)],
> hadoopConf: Configuration): Unit = {
> // Extract context and configuration from RDD.
> val sparkContext = rdd.context
> val stageId = rdd.id
> val sparkConf = rdd.conf
> val conf = new SerializableConfiguration(hadoopConf)
> // Set up a job.
> val jobTrackerId = SparkHadoopWriterUtils.createJobTrackerID(new Date())
> val jobAttemptId = new TaskAttemptID(jobTrackerId, stageId, TaskType.MAP,
> 0, 0)
> val jobContext = new TaskAttemptContextImpl(conf.value, jobAttemptId)
> val format = jobContext.getOutputFormatClass
> if (SparkHadoopWriterUtils.isOutputSpecValidationEnabled(sparkConf)) {
> // FileOutputFormat ignores the filesystem parameter
> val jobFormat = format.newInstance
> jobFormat.checkOutputSpecs(jobContext)
> }
> val committer = FileCommitProtocol.instantiate(
> className = classOf[HadoopMapReduceCommitProtocol].getName,
> jobId = stageId.toString,
> outputPath =
> conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
> isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
> committer.setupJob(jobContext)
> ...{code}
> in "write" function if output spec validation is enabled, it will call
> checkOutputSpec function in TableOutputFormat class, but the job format is
> simply created by "vall jobFormat = format.newInstance", this will NOT
> initialize "conf" member variable in TableOutputFormat class, let's continue
> check checkOutputSpecs function in TableOutputFormat class
>
> {code:java}
> // TableOutputFormat.checkOutputSpecs
> (org.apache.hadoop.hbase.mapreduce.TableOutputFormat.java) HBASE 1.4.0
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> try (Admin admin =
> ConnectionFactory.createConnection(getConf()).getAdmin()) {
> TableName tableName = TableName.valueOf(this.conf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameAsString());
> }
> if (!admin.isTableEnabled(tableName)) {
> throw new TableNotEnabledException("Can't write, table is not enabled:
> " +
> tableName.getNameAsString());
> }
> }
> }
> {code}
>
> "ConnectionFactory.createConnection(getConf())", as mentioned above "conf"
> class member is not initialized, so getConf() will return null, so in the
> next UserProvider create instance process, it throw the
> NullPointException(Please part of stack trace at the end), it is a little
> confused that, context passed by function parameter is actually been properly
> constructed, and it contains Configuration object, why context is never used?
> So I suggest to use below code
[jira] [Updated] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20520: -- Attachment: HBASE-20520.branch-2.0.002.patch > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20520: -- Attachment: HBASE-20520.branch-2.0.004.patch > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20520: -- Attachment: HBASE-20520.branch-2.0.003.patch > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461604#comment-16461604 ] stack commented on HBASE-20520: --- Add test to verify default takes in both WAL providers. Let me see how this does. Will be back with an edit to see if can clean up some of these stream constuctions... hard to follow what is going on . > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20295) TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
[
https://issues.apache.org/jira/browse/HBASE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461605#comment-16461605
]
Mike Drob commented on HBASE-20295:
---
My proposed patch (against branch-2.0) would look like this:
{noformat}
diff --git
a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
index 0a1928b21f..cdf6ccd9b3 100644
---
a/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
+++
b/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
@@ -174,9 +174,9 @@ implements Configurable {
@Override
public void checkOutputSpecs(JobContext context) throws IOException,
InterruptedException {
-Configuration hConf = context.getConfiguration();
+Configuration hConf = getConf();
if(hConf == null) {
- hConf = this.conf;
+ hConf = context.getConfiguration();
}
try (Admin admin = ConnectionFactory.createConnection(hConf).getAdmin()) {
{noformat}
But I don't know how to verify if this fix still works for the original spark
use case.
Probably need to fork off into a new issue since this has been released already.
> TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
> ---
>
> Key: HBASE-20295
> URL: https://issues.apache.org/jira/browse/HBASE-20295
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.4.0
> Environment: Spark 2.2.1, HBase 1.4.0
>Reporter: Michael Jin
>Assignee: Michael Jin
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20295.branch-1.4.001.patch,
> HBASE-20295.master.001.patch, HBASE-20295.master.002.patch,
> HBASE-20295.master.003.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I am using spark write data to HBase by using RDD.
> saveAsNewAPIHadoopDataset function, it works fine with hbase 1.3.1, but when
> update my hbase dependency to 1.4.0 in pom.xml, it throw
> java.lang.NullPointerException, it is caused by a logic error in
> TableOutputFormat.checkOutputSpecs function, please check below details:
> first let's take a look at SparkHadoopMapReduceWriter.write function in
> SparkHadoopMapReduceWriter.scala
> {code:java}
> // SparkHadoopMapReduceWriter.write
> (org.apache.spark.internal.io.SparkHadoopMapReduceWriter.scala)
> def write[K, V: ClassTag](
> rdd: RDD[(K, V)],
> hadoopConf: Configuration): Unit = {
> // Extract context and configuration from RDD.
> val sparkContext = rdd.context
> val stageId = rdd.id
> val sparkConf = rdd.conf
> val conf = new SerializableConfiguration(hadoopConf)
> // Set up a job.
> val jobTrackerId = SparkHadoopWriterUtils.createJobTrackerID(new Date())
> val jobAttemptId = new TaskAttemptID(jobTrackerId, stageId, TaskType.MAP,
> 0, 0)
> val jobContext = new TaskAttemptContextImpl(conf.value, jobAttemptId)
> val format = jobContext.getOutputFormatClass
> if (SparkHadoopWriterUtils.isOutputSpecValidationEnabled(sparkConf)) {
> // FileOutputFormat ignores the filesystem parameter
> val jobFormat = format.newInstance
> jobFormat.checkOutputSpecs(jobContext)
> }
> val committer = FileCommitProtocol.instantiate(
> className = classOf[HadoopMapReduceCommitProtocol].getName,
> jobId = stageId.toString,
> outputPath =
> conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
> isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
> committer.setupJob(jobContext)
> ...{code}
> in "write" function if output spec validation is enabled, it will call
> checkOutputSpec function in TableOutputFormat class, but the job format is
> simply created by "vall jobFormat = format.newInstance", this will NOT
> initialize "conf" member variable in TableOutputFormat class, let's continue
> check checkOutputSpecs function in TableOutputFormat class
>
> {code:java}
> // TableOutputFormat.checkOutputSpecs
> (org.apache.hadoop.hbase.mapreduce.TableOutputFormat.java) HBASE 1.4.0
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> try (Admin admin =
> ConnectionFactory.createConnection(getConf()).getAdmin()) {
> TableName tableName = TableName.valueOf(this.conf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameAsString());
> }
> if (!admin.isTableEnabled(tableName)) {
> throw new TableNotEnabledException("Can't write, table is not enabled:
> " +
> tableName.getNameAsString());
> }
> }
[jira] [Commented] (HBASE-20295) TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
[
https://issues.apache.org/jira/browse/HBASE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461670#comment-16461670
]
Michael Jin commented on HBASE-20295:
-
[~mdrob], sorry for the fails brought to you, I think checking sequence change
is OK for me, I will verify my original case in local, please wait a while, I
will update the verify result later.
> TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
> ---
>
> Key: HBASE-20295
> URL: https://issues.apache.org/jira/browse/HBASE-20295
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.4.0
> Environment: Spark 2.2.1, HBase 1.4.0
>Reporter: Michael Jin
>Assignee: Michael Jin
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20295.branch-1.4.001.patch,
> HBASE-20295.master.001.patch, HBASE-20295.master.002.patch,
> HBASE-20295.master.003.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I am using spark write data to HBase by using RDD.
> saveAsNewAPIHadoopDataset function, it works fine with hbase 1.3.1, but when
> update my hbase dependency to 1.4.0 in pom.xml, it throw
> java.lang.NullPointerException, it is caused by a logic error in
> TableOutputFormat.checkOutputSpecs function, please check below details:
> first let's take a look at SparkHadoopMapReduceWriter.write function in
> SparkHadoopMapReduceWriter.scala
> {code:java}
> // SparkHadoopMapReduceWriter.write
> (org.apache.spark.internal.io.SparkHadoopMapReduceWriter.scala)
> def write[K, V: ClassTag](
> rdd: RDD[(K, V)],
> hadoopConf: Configuration): Unit = {
> // Extract context and configuration from RDD.
> val sparkContext = rdd.context
> val stageId = rdd.id
> val sparkConf = rdd.conf
> val conf = new SerializableConfiguration(hadoopConf)
> // Set up a job.
> val jobTrackerId = SparkHadoopWriterUtils.createJobTrackerID(new Date())
> val jobAttemptId = new TaskAttemptID(jobTrackerId, stageId, TaskType.MAP,
> 0, 0)
> val jobContext = new TaskAttemptContextImpl(conf.value, jobAttemptId)
> val format = jobContext.getOutputFormatClass
> if (SparkHadoopWriterUtils.isOutputSpecValidationEnabled(sparkConf)) {
> // FileOutputFormat ignores the filesystem parameter
> val jobFormat = format.newInstance
> jobFormat.checkOutputSpecs(jobContext)
> }
> val committer = FileCommitProtocol.instantiate(
> className = classOf[HadoopMapReduceCommitProtocol].getName,
> jobId = stageId.toString,
> outputPath =
> conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
> isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
> committer.setupJob(jobContext)
> ...{code}
> in "write" function if output spec validation is enabled, it will call
> checkOutputSpec function in TableOutputFormat class, but the job format is
> simply created by "vall jobFormat = format.newInstance", this will NOT
> initialize "conf" member variable in TableOutputFormat class, let's continue
> check checkOutputSpecs function in TableOutputFormat class
>
> {code:java}
> // TableOutputFormat.checkOutputSpecs
> (org.apache.hadoop.hbase.mapreduce.TableOutputFormat.java) HBASE 1.4.0
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> try (Admin admin =
> ConnectionFactory.createConnection(getConf()).getAdmin()) {
> TableName tableName = TableName.valueOf(this.conf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameAsString());
> }
> if (!admin.isTableEnabled(tableName)) {
> throw new TableNotEnabledException("Can't write, table is not enabled:
> " +
> tableName.getNameAsString());
> }
> }
> }
> {code}
>
> "ConnectionFactory.createConnection(getConf())", as mentioned above "conf"
> class member is not initialized, so getConf() will return null, so in the
> next UserProvider create instance process, it throw the
> NullPointException(Please part of stack trace at the end), it is a little
> confused that, context passed by function parameter is actually been properly
> constructed, and it contains Configuration object, why context is never used?
> So I suggest to use below code to partly fix this issue:
>
> {code:java}
> // code placeholder
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> Configuration hConf = context.getConfiguration();
> if(hConf == null)
> hConf = this.conf;
> try (Admin admin = ConnectionFactory.createConnection(hConf).getAdmin()) {
> TableName tableName = TableName.valueOf(
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[
https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461702#comment-16461702
]
Hadoop QA commented on HBASE-20520:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
15s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
31s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} branch-2.0 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
12s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
32s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} hbase-server: The patch generated 5 new + 51 unchanged
- 1 fixed = 56 total (was 52) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
59s{color} | {color:red} root: The patch generated 5 new + 51 unchanged - 1
fixed = 56 total (was 52) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
50s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 32m 37s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:bla
[jira] [Updated] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20169:
--
Resolution: Fixed
Assignee: Chia-Ping Tsai (was: stack)
Hadoop Flags: Reviewed
Fix Version/s: 2.0.1
3.0.0
Status: Resolved (was: Patch Available)
Pushed to branch-2.0+. Assigned [~chia7712] because he did the hard work.
Thanks for reviews.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 3.0.0, 2.0.1
>
> Attachments: HBASE-20169.branch-2.0.001.patch,
> HBASE-20169.branch-2.001.patch, HBASE-20169.branch-2.002.patch,
> HBASE-20169.branch-2.003.patch, HBASE-20169.branch-2.004.patch,
> HBASE-20169.branch-2.005.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20295) TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
[
https://issues.apache.org/jira/browse/HBASE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461740#comment-16461740
]
Michael Jin commented on HBASE-20295:
-
[~mdrob], just verified in local, it works from my side
[~yuzhih...@gmail.com], [~elserj], [~stack], fork a new issue or just add a new
patch in this thread, how about your guys idea?
> TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
> ---
>
> Key: HBASE-20295
> URL: https://issues.apache.org/jira/browse/HBASE-20295
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.4.0
> Environment: Spark 2.2.1, HBase 1.4.0
>Reporter: Michael Jin
>Assignee: Michael Jin
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20295.branch-1.4.001.patch,
> HBASE-20295.master.001.patch, HBASE-20295.master.002.patch,
> HBASE-20295.master.003.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I am using spark write data to HBase by using RDD.
> saveAsNewAPIHadoopDataset function, it works fine with hbase 1.3.1, but when
> update my hbase dependency to 1.4.0 in pom.xml, it throw
> java.lang.NullPointerException, it is caused by a logic error in
> TableOutputFormat.checkOutputSpecs function, please check below details:
> first let's take a look at SparkHadoopMapReduceWriter.write function in
> SparkHadoopMapReduceWriter.scala
> {code:java}
> // SparkHadoopMapReduceWriter.write
> (org.apache.spark.internal.io.SparkHadoopMapReduceWriter.scala)
> def write[K, V: ClassTag](
> rdd: RDD[(K, V)],
> hadoopConf: Configuration): Unit = {
> // Extract context and configuration from RDD.
> val sparkContext = rdd.context
> val stageId = rdd.id
> val sparkConf = rdd.conf
> val conf = new SerializableConfiguration(hadoopConf)
> // Set up a job.
> val jobTrackerId = SparkHadoopWriterUtils.createJobTrackerID(new Date())
> val jobAttemptId = new TaskAttemptID(jobTrackerId, stageId, TaskType.MAP,
> 0, 0)
> val jobContext = new TaskAttemptContextImpl(conf.value, jobAttemptId)
> val format = jobContext.getOutputFormatClass
> if (SparkHadoopWriterUtils.isOutputSpecValidationEnabled(sparkConf)) {
> // FileOutputFormat ignores the filesystem parameter
> val jobFormat = format.newInstance
> jobFormat.checkOutputSpecs(jobContext)
> }
> val committer = FileCommitProtocol.instantiate(
> className = classOf[HadoopMapReduceCommitProtocol].getName,
> jobId = stageId.toString,
> outputPath =
> conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
> isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
> committer.setupJob(jobContext)
> ...{code}
> in "write" function if output spec validation is enabled, it will call
> checkOutputSpec function in TableOutputFormat class, but the job format is
> simply created by "vall jobFormat = format.newInstance", this will NOT
> initialize "conf" member variable in TableOutputFormat class, let's continue
> check checkOutputSpecs function in TableOutputFormat class
>
> {code:java}
> // TableOutputFormat.checkOutputSpecs
> (org.apache.hadoop.hbase.mapreduce.TableOutputFormat.java) HBASE 1.4.0
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> try (Admin admin =
> ConnectionFactory.createConnection(getConf()).getAdmin()) {
> TableName tableName = TableName.valueOf(this.conf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameAsString());
> }
> if (!admin.isTableEnabled(tableName)) {
> throw new TableNotEnabledException("Can't write, table is not enabled:
> " +
> tableName.getNameAsString());
> }
> }
> }
> {code}
>
> "ConnectionFactory.createConnection(getConf())", as mentioned above "conf"
> class member is not initialized, so getConf() will return null, so in the
> next UserProvider create instance process, it throw the
> NullPointException(Please part of stack trace at the end), it is a little
> confused that, context passed by function parameter is actually been properly
> constructed, and it contains Configuration object, why context is never used?
> So I suggest to use below code to partly fix this issue:
>
> {code:java}
> // code placeholder
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> Configuration hConf = context.getConfiguration();
> if(hConf == null)
> hConf = this.conf;
> try (Admin admin = ConnectionFactory.createConnection(hConf).getAdmin()) {
> TableName tableName = TableName.valueOf(hConf.get(
[jira] [Commented] (HBASE-20295) TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
[
https://issues.apache.org/jira/browse/HBASE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461747#comment-16461747
]
Ted Yu commented on HBASE-20295:
Please open new issue.
> TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
> ---
>
> Key: HBASE-20295
> URL: https://issues.apache.org/jira/browse/HBASE-20295
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.4.0
> Environment: Spark 2.2.1, HBase 1.4.0
>Reporter: Michael Jin
>Assignee: Michael Jin
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20295.branch-1.4.001.patch,
> HBASE-20295.master.001.patch, HBASE-20295.master.002.patch,
> HBASE-20295.master.003.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I am using spark write data to HBase by using RDD.
> saveAsNewAPIHadoopDataset function, it works fine with hbase 1.3.1, but when
> update my hbase dependency to 1.4.0 in pom.xml, it throw
> java.lang.NullPointerException, it is caused by a logic error in
> TableOutputFormat.checkOutputSpecs function, please check below details:
> first let's take a look at SparkHadoopMapReduceWriter.write function in
> SparkHadoopMapReduceWriter.scala
> {code:java}
> // SparkHadoopMapReduceWriter.write
> (org.apache.spark.internal.io.SparkHadoopMapReduceWriter.scala)
> def write[K, V: ClassTag](
> rdd: RDD[(K, V)],
> hadoopConf: Configuration): Unit = {
> // Extract context and configuration from RDD.
> val sparkContext = rdd.context
> val stageId = rdd.id
> val sparkConf = rdd.conf
> val conf = new SerializableConfiguration(hadoopConf)
> // Set up a job.
> val jobTrackerId = SparkHadoopWriterUtils.createJobTrackerID(new Date())
> val jobAttemptId = new TaskAttemptID(jobTrackerId, stageId, TaskType.MAP,
> 0, 0)
> val jobContext = new TaskAttemptContextImpl(conf.value, jobAttemptId)
> val format = jobContext.getOutputFormatClass
> if (SparkHadoopWriterUtils.isOutputSpecValidationEnabled(sparkConf)) {
> // FileOutputFormat ignores the filesystem parameter
> val jobFormat = format.newInstance
> jobFormat.checkOutputSpecs(jobContext)
> }
> val committer = FileCommitProtocol.instantiate(
> className = classOf[HadoopMapReduceCommitProtocol].getName,
> jobId = stageId.toString,
> outputPath =
> conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
> isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
> committer.setupJob(jobContext)
> ...{code}
> in "write" function if output spec validation is enabled, it will call
> checkOutputSpec function in TableOutputFormat class, but the job format is
> simply created by "vall jobFormat = format.newInstance", this will NOT
> initialize "conf" member variable in TableOutputFormat class, let's continue
> check checkOutputSpecs function in TableOutputFormat class
>
> {code:java}
> // TableOutputFormat.checkOutputSpecs
> (org.apache.hadoop.hbase.mapreduce.TableOutputFormat.java) HBASE 1.4.0
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> try (Admin admin =
> ConnectionFactory.createConnection(getConf()).getAdmin()) {
> TableName tableName = TableName.valueOf(this.conf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameAsString());
> }
> if (!admin.isTableEnabled(tableName)) {
> throw new TableNotEnabledException("Can't write, table is not enabled:
> " +
> tableName.getNameAsString());
> }
> }
> }
> {code}
>
> "ConnectionFactory.createConnection(getConf())", as mentioned above "conf"
> class member is not initialized, so getConf() will return null, so in the
> next UserProvider create instance process, it throw the
> NullPointException(Please part of stack trace at the end), it is a little
> confused that, context passed by function parameter is actually been properly
> constructed, and it contains Configuration object, why context is never used?
> So I suggest to use below code to partly fix this issue:
>
> {code:java}
> // code placeholder
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> Configuration hConf = context.getConfiguration();
> if(hConf == null)
> hConf = this.conf;
> try (Admin admin = ConnectionFactory.createConnection(hConf).getAdmin()) {
> TableName tableName = TableName.valueOf(hConf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameA
[jira] [Created] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails
Michael Jin created HBASE-20521: --- Summary: TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails Key: HBASE-20521 URL: https://issues.apache.org/jira/browse/HBASE-20521 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 2.0.0 Environment: spark 2.2.1, hbase 2.0.0 Reporter: Michael Jin Assignee: Michael Jin HBASE-20295 fix null point exception of "conf" member variable, add "context.getConfiguration()" in case when "conf" object was not been properly initialized, and put it into the first priority checking sequence, this code change affect user call "setConf" explicitly initialize "conf" object in TableOutputFormat object, proposal to change checking sequence, use "conf" object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20295) TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
[
https://issues.apache.org/jira/browse/HBASE-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461775#comment-16461775
]
Michael Jin commented on HBASE-20295:
-
new issue was created: HBASE-20521
[~mdrob], please correct description if it's not clear
> TableOutputFormat.checkOutputSpecs throw NullPointerException Exception
> ---
>
> Key: HBASE-20295
> URL: https://issues.apache.org/jira/browse/HBASE-20295
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 1.4.0
> Environment: Spark 2.2.1, HBase 1.4.0
>Reporter: Michael Jin
>Assignee: Michael Jin
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20295.branch-1.4.001.patch,
> HBASE-20295.master.001.patch, HBASE-20295.master.002.patch,
> HBASE-20295.master.003.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I am using spark write data to HBase by using RDD.
> saveAsNewAPIHadoopDataset function, it works fine with hbase 1.3.1, but when
> update my hbase dependency to 1.4.0 in pom.xml, it throw
> java.lang.NullPointerException, it is caused by a logic error in
> TableOutputFormat.checkOutputSpecs function, please check below details:
> first let's take a look at SparkHadoopMapReduceWriter.write function in
> SparkHadoopMapReduceWriter.scala
> {code:java}
> // SparkHadoopMapReduceWriter.write
> (org.apache.spark.internal.io.SparkHadoopMapReduceWriter.scala)
> def write[K, V: ClassTag](
> rdd: RDD[(K, V)],
> hadoopConf: Configuration): Unit = {
> // Extract context and configuration from RDD.
> val sparkContext = rdd.context
> val stageId = rdd.id
> val sparkConf = rdd.conf
> val conf = new SerializableConfiguration(hadoopConf)
> // Set up a job.
> val jobTrackerId = SparkHadoopWriterUtils.createJobTrackerID(new Date())
> val jobAttemptId = new TaskAttemptID(jobTrackerId, stageId, TaskType.MAP,
> 0, 0)
> val jobContext = new TaskAttemptContextImpl(conf.value, jobAttemptId)
> val format = jobContext.getOutputFormatClass
> if (SparkHadoopWriterUtils.isOutputSpecValidationEnabled(sparkConf)) {
> // FileOutputFormat ignores the filesystem parameter
> val jobFormat = format.newInstance
> jobFormat.checkOutputSpecs(jobContext)
> }
> val committer = FileCommitProtocol.instantiate(
> className = classOf[HadoopMapReduceCommitProtocol].getName,
> jobId = stageId.toString,
> outputPath =
> conf.value.get("mapreduce.output.fileoutputformat.outputdir"),
> isAppend = false).asInstanceOf[HadoopMapReduceCommitProtocol]
> committer.setupJob(jobContext)
> ...{code}
> in "write" function if output spec validation is enabled, it will call
> checkOutputSpec function in TableOutputFormat class, but the job format is
> simply created by "vall jobFormat = format.newInstance", this will NOT
> initialize "conf" member variable in TableOutputFormat class, let's continue
> check checkOutputSpecs function in TableOutputFormat class
>
> {code:java}
> // TableOutputFormat.checkOutputSpecs
> (org.apache.hadoop.hbase.mapreduce.TableOutputFormat.java) HBASE 1.4.0
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> try (Admin admin =
> ConnectionFactory.createConnection(getConf()).getAdmin()) {
> TableName tableName = TableName.valueOf(this.conf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundException("Can't write, table does not exist:" +
> tableName.getNameAsString());
> }
> if (!admin.isTableEnabled(tableName)) {
> throw new TableNotEnabledException("Can't write, table is not enabled:
> " +
> tableName.getNameAsString());
> }
> }
> }
> {code}
>
> "ConnectionFactory.createConnection(getConf())", as mentioned above "conf"
> class member is not initialized, so getConf() will return null, so in the
> next UserProvider create instance process, it throw the
> NullPointException(Please part of stack trace at the end), it is a little
> confused that, context passed by function parameter is actually been properly
> constructed, and it contains Configuration object, why context is never used?
> So I suggest to use below code to partly fix this issue:
>
> {code:java}
> // code placeholder
> @Override
> public void checkOutputSpecs(JobContext context) throws IOException,
> InterruptedException {
> Configuration hConf = context.getConfiguration();
> if(hConf == null)
> hConf = this.conf;
> try (Admin admin = ConnectionFactory.createConnection(hConf).getAdmin()) {
> TableName tableName = TableName.valueOf(hConf.get(OUTPUT_TABLE));
> if (!admin.tableExists(tableName)) {
> throw new TableNotFoundExcep
[jira] [Commented] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails
[ https://issues.apache.org/jira/browse/HBASE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461786#comment-16461786 ] Mike Drob commented on HBASE-20521: --- Is it possibly to include a unit test for the spark case in this, or is that going to be very hard and take lots of scaffolding? > TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script > fails > > > Key: HBASE-20521 > URL: https://issues.apache.org/jira/browse/HBASE-20521 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0 > Environment: spark 2.2.1, hbase 2.0.0 >Reporter: Michael Jin >Assignee: Michael Jin >Priority: Major > > HBASE-20295 fix null point exception of "conf" member variable, add > "context.getConfiguration()" in case when "conf" object was not been properly > initialized, and put it into the first priority checking sequence, this code > change affect user call "setConf" explicitly initialize "conf" object in > TableOutputFormat object, proposal to change checking sequence, use "conf" > object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails
[ https://issues.apache.org/jira/browse/HBASE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461796#comment-16461796 ] Michael Jin commented on HBASE-20521: - [~mdrob], good suggestion, let me check if I can add UT in hbase-spark module, it's not that a easy task before 2.0 since there is no native support for spark on hbase, I will reply you after take a look at hbase-spark module. > TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script > fails > > > Key: HBASE-20521 > URL: https://issues.apache.org/jira/browse/HBASE-20521 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0 > Environment: spark 2.2.1, hbase 2.0.0 >Reporter: Michael Jin >Assignee: Michael Jin >Priority: Major > > HBASE-20295 fix null point exception of "conf" member variable, add > "context.getConfiguration()" in case when "conf" object was not been properly > initialized, and put it into the first priority checking sequence, this code > change affect user call "setConf" explicitly initialize "conf" object in > TableOutputFormat object, proposal to change checking sequence, use "conf" > object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails
[ https://issues.apache.org/jira/browse/HBASE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20521: -- Fix Version/s: 2.0.1 2.1.0 3.0.0 > TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script > fails > > > Key: HBASE-20521 > URL: https://issues.apache.org/jira/browse/HBASE-20521 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0 > Environment: spark 2.2.1, hbase 2.0.0 >Reporter: Michael Jin >Assignee: Michael Jin >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > > HBASE-20295 fix null point exception of "conf" member variable, add > "context.getConfiguration()" in case when "conf" object was not been properly > initialized, and put it into the first priority checking sequence, this code > change affect user call "setConf" explicitly initialize "conf" object in > TableOutputFormat object, proposal to change checking sequence, use "conf" > object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20520: -- Attachment: HBASE-20520.branch-2.0.005.patch > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch, HBASE-20520.branch-2.0.005.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461821#comment-16461821 ] stack commented on HBASE-20520: --- .005 Edit. Fix test. > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch, HBASE-20520.branch-2.0.005.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails
[ https://issues.apache.org/jira/browse/HBASE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461828#comment-16461828 ] Mike Drob commented on HBASE-20521: --- 2.0 doesn't have spark support either, but the hbase-spark module is in our 'master' branch, so you can look there. > TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script > fails > > > Key: HBASE-20521 > URL: https://issues.apache.org/jira/browse/HBASE-20521 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0 > Environment: spark 2.2.1, hbase 2.0.0 >Reporter: Michael Jin >Assignee: Michael Jin >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > > HBASE-20295 fix null point exception of "conf" member variable, add > "context.getConfiguration()" in case when "conf" object was not been properly > initialized, and put it into the first priority checking sequence, this code > change affect user call "setConf" explicitly initialize "conf" object in > TableOutputFormat object, proposal to change checking sequence, use "conf" > object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20368) Fix RIT stuck when a rsgroup has no online servers but AM's pendingAssginQueue is cleared
[ https://issues.apache.org/jira/browse/HBASE-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461832#comment-16461832 ] Xiaolin Ha commented on HBASE-20368: ping [~mdrob] > Fix RIT stuck when a rsgroup has no online servers but AM's > pendingAssginQueue is cleared > - > > Key: HBASE-20368 > URL: https://issues.apache.org/jira/browse/HBASE-20368 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.0 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: HBASE-20368.branch-2.001.patch, > HBASE-20368.branch-2.002.patch, HBASE-20368.branch-2.003.patch > > > This error can be reproduced by shutting down all servers in a rsgroups and > starting them soon afterwards. > The regions on this rsgroup will be reassigned, but there is no available > servers of this rsgroup. > They will be added to AM's pendingAssginQueue, which AM will clear regardless > of the result of assigning in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461842#comment-16461842 ] Mike Drob commented on HBASE-20520: --- change makes sense, v5 LGTM. +1 assuming QA is happy. > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch, HBASE-20520.branch-2.0.005.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20481) Add a setSerial method to ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461860#comment-16461860 ] Zheng Hu commented on HBASE-20481: -- bq. so the current API is enough to tell the ReplicationEndpoint whether the peer is serial or not... OK, the setSerial() method can be removed now.. bq. You make everything serial, but actually, we only need the serial guarantee inside a region, for different regions we can replicate them parallelly... You are right, will update this patch.. > Add a setSerial method to ReplicationEndpoint > - > > Key: HBASE-20481 > URL: https://issues.apache.org/jira/browse/HBASE-20481 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-20481.v1.patch > > > When debugging HBASE-20475, [~openinx] found that the > HBaseInterClusterReplicationEndpoint may send the entries for the same > regions concurrently, which breaks the serial replication. > As long as we can have multiple ReplicationEndpoint implementation, just fix > HBaseInterClusterReplicationEndpoint is not enough, we need to add a > setSerial method to ReplicationEndpoint, to tell the implementation that you > should keep the order of the entries from the same region. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20522) Maven artifacts for HBase 2 are not available at maven central
Ismaël Mejía created HBASE-20522: Summary: Maven artifacts for HBase 2 are not available at maven central Key: HBASE-20522 URL: https://issues.apache.org/jira/browse/HBASE-20522 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Ismaël Mejía -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20520) Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize
[ https://issues.apache.org/jira/browse/HBASE-20520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461893#comment-16461893 ] stack commented on HBASE-20520: --- Thanks [~mdrob] I got a little lost in the hierarchy here. We should do a bit of a refactor one day > Failed effort upping default HDFS blocksize, hbase.regionserver.hlog.blocksize > -- > > Key: HBASE-20520 > URL: https://issues.apache.org/jira/browse/HBASE-20520 > Project: HBase > Issue Type: Sub-task > Components: conf >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.1 > > Attachments: HBASE-20520.branch-2.0.001.patch, > HBASE-20520.branch-2.0.002.patch, HBASE-20520.branch-2.0.003.patch, > HBASE-20520.branch-2.0.004.patch, HBASE-20520.branch-2.0.005.patch > > > Good one found by our [~mdrob]. Problem in the parent issue where failed > attempt doubling default block size but halving the size at which roll -- so > we end up in roughly same place only we make less small WALs in those cases > where we are writing furiously and fail to roll before starting a new block. > From Drob: ".../we drop default log roll to 0.5, but we don't actually > increase the block size... AbstractProtobufLogWriter.java:164 still uses > default HDFS block sizing and AbstractFSWAL.java:408 block size is only used > in a log message, never actually makes it to files...". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[ https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461907#comment-16461907 ] Xiaolin Ha commented on HBASE-19997: [~stack]When master rolling from 0.98 to 2.0, 0.98 regionserver open hbase:meta throws an NPE. Will this problem be solved? I have made some testing work on rolling from 0.98 to 2.0 and found that, firstly rolling update regionservers and secondly rolling update masters will be OK. Another problem is, will the added 'table' columnfamily make incompatible of 0.98 client working with hbase2 ? > [rolling upgrade] 1.x => 2.x > > > Key: HBASE-19997 > URL: https://issues.apache.org/jira/browse/HBASE-19997 > Project: HBase > Issue Type: Umbrella >Reporter: stack >Priority: Blocker > Fix For: 2.1.0 > > > An umbrella issue of issues needed so folks can do a rolling upgrade from > hbase-1.x to hbase-2.x. > (Recent) Notables: > * hbase-1.x can't read hbase-2.x WALs -- hbase-1.x doesn't know the > AsyncProtobufLogWriter class used writing the WAL -- see > https://issues.apache.org/jira/browse/HBASE-19166?focusedCommentId=16362897&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16362897 > for exception. > ** Might be ok... means WAL split fails on an hbase1 RS... must wait till an > hbase-2.x RS picks up the WAL for it to be split. > * hbase-1 can't open regions from tables created by hbase-2; it can't find > the Table descriptor. See > https://issues.apache.org/jira/browse/HBASE-19116?focusedCommentId=16363276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16363276 > ** This might be ok if the tables we are doing rolling upgrade over were > written with hbase-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[ https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461913#comment-16461913 ] stack commented on HBASE-19997: --- Hello [~Xiaolin Ha] Thanks for taking a look at this one. On the NPE, I hacked something. Don't remember what. It had to do w/ startup sequence in Master. What is the exception on the regionserver side? That might remind me. Why you doing RS first and then Master? Does it work? I'd think you'd have to do Master first no? Maybe you have a better idea? Coming up from 0.98, you'd need to figure a solution for HBASE-20237. Old clients will sometimes fail finding a region in hbase:meta. See comments on that issue. Ask again if not clear. > [rolling upgrade] 1.x => 2.x > > > Key: HBASE-19997 > URL: https://issues.apache.org/jira/browse/HBASE-19997 > Project: HBase > Issue Type: Umbrella >Reporter: stack >Priority: Blocker > Fix For: 2.1.0 > > > An umbrella issue of issues needed so folks can do a rolling upgrade from > hbase-1.x to hbase-2.x. > (Recent) Notables: > * hbase-1.x can't read hbase-2.x WALs -- hbase-1.x doesn't know the > AsyncProtobufLogWriter class used writing the WAL -- see > https://issues.apache.org/jira/browse/HBASE-19166?focusedCommentId=16362897&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16362897 > for exception. > ** Might be ok... means WAL split fails on an hbase1 RS... must wait till an > hbase-2.x RS picks up the WAL for it to be split. > * hbase-1 can't open regions from tables created by hbase-2; it can't find > the Table descriptor. See > https://issues.apache.org/jira/browse/HBASE-19116?focusedCommentId=16363276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16363276 > ** This might be ok if the tables we are doing rolling upgrade over were > written with hbase-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461925#comment-16461925
]
Hudson commented on HBASE-20169:
Results for branch branch-2
[build #688 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 3.0.0, 2.0.1
>
> Attachments: HBASE-20169.branch-2.0.001.patch,
> HBASE-20169.branch-2.001.patch, HBASE-20169.branch-2.002.patch,
> HBASE-20169.branch-2.003.patch, HBASE-20169.branch-2.004.patch,
> HBASE-20169.branch-2.005.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20514) On Master restart if table is stuck in DISABLING state, CLOSED regions should not be considered stuck in-transition
[
https://issues.apache.org/jira/browse/HBASE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461924#comment-16461924
]
Hudson commented on HBASE-20514:
Results for branch branch-2
[build #688 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/688//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> On Master restart if table is stuck in DISABLING state, CLOSED regions should
> not be considered stuck in-transition
> ---
>
> Key: HBASE-20514
> URL: https://issues.apache.org/jira/browse/HBASE-20514
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.0.0
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: hbase-20514.master.001.patch
>
>
> When master is restarted, in AssignmentManager#loadMeta(), if table is in
> DISABLED state nothing is done for regions in CLOSED state. But if table is
> stuck in DISABLING state then CLOSED regions are considered as stuck
> in-transition. CLOSED regions of DISABLING/ DISABLED table can be handled the
> same way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails
[ https://issues.apache.org/jira/browse/HBASE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461936#comment-16461936 ] Michael Jin commented on HBASE-20521: - [~mdrob], I think it's possible to add a unit test into "hbase-spark" module, need sometime to coding and testing, I will attach patch and let you review. > TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script > fails > > > Key: HBASE-20521 > URL: https://issues.apache.org/jira/browse/HBASE-20521 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0 > Environment: spark 2.2.1, hbase 2.0.0 >Reporter: Michael Jin >Assignee: Michael Jin >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > > HBASE-20295 fix null point exception of "conf" member variable, add > "context.getConfiguration()" in case when "conf" object was not been properly > initialized, and put it into the first priority checking sequence, this code > change affect user call "setConf" explicitly initialize "conf" object in > TableOutputFormat object, proposal to change checking sequence, use "conf" > object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20521) TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script run fail
[ https://issues.apache.org/jira/browse/HBASE-20521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Jin updated HBASE-20521: Summary: TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script run fail (was: TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script fails) > TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script > run fail > --- > > Key: HBASE-20521 > URL: https://issues.apache.org/jira/browse/HBASE-20521 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 2.0.0 > Environment: spark 2.2.1, hbase 2.0.0 >Reporter: Michael Jin >Assignee: Michael Jin >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > > HBASE-20295 fix null point exception of "conf" member variable, add > "context.getConfiguration()" in case when "conf" object was not been properly > initialized, and put it into the first priority checking sequence, this code > change affect user call "setConf" explicitly initialize "conf" object in > TableOutputFormat object, proposal to change checking sequence, use "conf" > object from "getConf" method first . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20523) PE tool should support configuring client side buffering sizes
ramkrishna.s.vasudevan created HBASE-20523: -- Summary: PE tool should support configuring client side buffering sizes Key: HBASE-20523 URL: https://issues.apache.org/jira/browse/HBASE-20523 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.1.0, 2.0.1 The client side buffering size impacts the write load and the write performance. Hence for testing purpose it is better we allow client side buffering to be configurable in PE. Already YCSB has such facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20169) NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey)
[
https://issues.apache.org/jira/browse/HBASE-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461945#comment-16461945
]
Hudson commented on HBASE-20169:
Results for branch branch-2.0
[build #252 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is
> flakey)
> --
>
> Key: HBASE-20169
> URL: https://issues.apache.org/jira/browse/HBASE-20169
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Duo Zhang
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 3.0.0, 2.0.1
>
> Attachments: HBASE-20169.branch-2.0.001.patch,
> HBASE-20169.branch-2.001.patch, HBASE-20169.branch-2.002.patch,
> HBASE-20169.branch-2.003.patch, HBASE-20169.branch-2.004.patch,
> HBASE-20169.branch-2.005.patch, HBASE-20169.master.001.ADDENDUM.patch,
> HBASE-20169.master.001.ADDENDUM.patch, HBASE-20169.v0.addendum.patch
>
>
> This usually happens when some master or rs has already been down before we
> calling shutdownMiniCluster.
> See
> https://builds.apache.org/job/HBASE-Flaky-Tests/27223/testReport/junit/org.apache.hadoop.hbase.master/TestAssignmentManagerMetrics/org_apache_hadoop_hbase_master_TestAssignmentManagerMetrics/
> and also
> http://104.198.223.121:8080/job/HBASE-Flaky-Tests/34873/testReport/junit/org.apache.hadoop.hbase.master/TestRestartCluster/testRetainAssignmentOnRestart/
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.after(TestAssignmentManagerMetrics.java:100)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.master.TestRestartCluster.testRetainAssignmentOnRestart(TestRestartCluster.java:156)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20514) On Master restart if table is stuck in DISABLING state, CLOSED regions should not be considered stuck in-transition
[
https://issues.apache.org/jira/browse/HBASE-20514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461944#comment-16461944
]
Hudson commented on HBASE-20514:
Results for branch branch-2.0
[build #252 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/252//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> On Master restart if table is stuck in DISABLING state, CLOSED regions should
> not be considered stuck in-transition
> ---
>
> Key: HBASE-20514
> URL: https://issues.apache.org/jira/browse/HBASE-20514
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.0.0
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: hbase-20514.master.001.patch
>
>
> When master is restarted, in AssignmentManager#loadMeta(), if table is in
> DISABLED state nothing is done for regions in CLOSED state. But if table is
> stuck in DISABLING state then CLOSED regions are considered as stuck
> in-transition. CLOSED regions of DISABLING/ DISABLED table can be handled the
> same way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20523) PE tool should support configuring client side buffering sizes
[ https://issues.apache.org/jira/browse/HBASE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-20523: --- Priority: Minor (was: Major) > PE tool should support configuring client side buffering sizes > -- > > Key: HBASE-20523 > URL: https://issues.apache.org/jira/browse/HBASE-20523 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.1.0, 2.0.1 > > > The client side buffering size impacts the write load and the write > performance. Hence for testing purpose it is better we allow client side > buffering to be configurable in PE. Already YCSB has such facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20523) PE tool should support configuring client side buffering sizes
[ https://issues.apache.org/jira/browse/HBASE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462000#comment-16462000 ] ramkrishna.s.vasudevan commented on HBASE-20523: Patch based on branch-2. Tested the patch and it works. > PE tool should support configuring client side buffering sizes > -- > > Key: HBASE-20523 > URL: https://issues.apache.org/jira/browse/HBASE-20523 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.1.0, 2.0.1 > > Attachments: HBASE-20523.patch > > > The client side buffering size impacts the write load and the write > performance. Hence for testing purpose it is better we allow client side > buffering to be configurable in PE. Already YCSB has such facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20523) PE tool should support configuring client side buffering sizes
[ https://issues.apache.org/jira/browse/HBASE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-20523: --- Attachment: HBASE-20523.patch > PE tool should support configuring client side buffering sizes > -- > > Key: HBASE-20523 > URL: https://issues.apache.org/jira/browse/HBASE-20523 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.1.0, 2.0.1 > > Attachments: HBASE-20523.patch > > > The client side buffering size impacts the write load and the write > performance. Hence for testing purpose it is better we allow client side > buffering to be configurable in PE. Already YCSB has such facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20281) [DOC] upgrade section needs an explanation of changes to Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462007#comment-16462007
]
Anoop Sam John commented on HBASE-20281:
The config related to combined mode is still valid for 1.x. That is why kept it
under pre 2.0 version.
Heap mode is removed as it is not really useful. We save nothing there. It
makes the xmx need still very high. So instead of BC one can use LRU cache ic
xmx is not an issue. I came across a study report comparing off heap and on
heap BC. (File mode also) will post that link also here.
> [DOC] upgrade section needs an explanation of changes to Bucket Cache
> -
>
> Key: HBASE-20281
> URL: https://issues.apache.org/jira/browse/HBASE-20281
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache, documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> the first pass version of the upgrade docs has a brief note about the bucket
> cache changing:
> {quote}
> * hbase.bucketcache.combinedcache.enabled
> * hbase.bucketcache.ioengine no longer supports the 'heap' value.
> {quote}
> But the changes are substantial and warrant a section of their own under the
> "Changes of Note!" banner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[ https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaolin Ha updated HBASE-19997: --- Attachment: Screenshot from 2018-05-03 14-43-46.png > [rolling upgrade] 1.x => 2.x > > > Key: HBASE-19997 > URL: https://issues.apache.org/jira/browse/HBASE-19997 > Project: HBase > Issue Type: Umbrella >Reporter: stack >Priority: Blocker > Fix For: 2.1.0 > > Attachments: Screenshot from 2018-05-03 14-43-46.png > > > An umbrella issue of issues needed so folks can do a rolling upgrade from > hbase-1.x to hbase-2.x. > (Recent) Notables: > * hbase-1.x can't read hbase-2.x WALs -- hbase-1.x doesn't know the > AsyncProtobufLogWriter class used writing the WAL -- see > https://issues.apache.org/jira/browse/HBASE-19166?focusedCommentId=16362897&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16362897 > for exception. > ** Might be ok... means WAL split fails on an hbase1 RS... must wait till an > hbase-2.x RS picks up the WAL for it to be split. > * hbase-1 can't open regions from tables created by hbase-2; it can't find > the Table descriptor. See > https://issues.apache.org/jira/browse/HBASE-19116?focusedCommentId=16363276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16363276 > ** This might be ok if the tables we are doing rolling upgrade over were > written with hbase-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[ https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462009#comment-16462009 ] Xiaolin Ha commented on HBASE-19997: Hello [~stack] When rolling update masters from 0.98 to 2.0 while regionservers are 0.98, master will encounter NPE as well as regionserver. Logs on regionserver is, 2018-05-03,11:46:47,996 ERROR [PriorityRpcServer.handler=0,queue=0,port=37900] org.apache.hadoop.ipc.RpcServer: Unexpected throwable object java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:4807) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:21048) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128) at java.lang.Thread.run(Thread.java:745) 2018-05-03,11:46:48,335 ERROR [PriorityRpcServer.handler=1,queue=0,port=37900] org.apache.hadoop.ipc.RpcServer: Unexpected throwable object java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:4807) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:21048) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128) at java.lang.Thread.run(Thread.java:745) and logs on master is, 2018-05-03,11:16:34,930 INFO [PEWorker-3] org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=c4-hadoop-tst-st28.bj,37900,1525316678267; rit=OPENING, location=c4-hadoop-tst-st28.bj,37900,1525316678267 2018-05-03,11:16:35,081 INFO [ProcedureDispatcherTimeoutThread] org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Fallback to compat rpc execution for serverName=c4-hadoop-tst-st28.bj,37900,1525316678267 version=401419 2018-05-03,11:16:35,193 WARN [RSProcedureDispatcher-pool3-t2] org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher: Failed dispatch to server=c4-hadoop-tst-st28.bj,37900,1525316678267 try=0 java.io.IOException: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2094) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:4807) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:21048) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061) ... 4 more @c4-hadoop-tst-st28.bj/10.132.2.27:37900 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:358) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:335) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$OpenRegionRemoteCall.sendRequest(RSProcedureDispatcher.java:392) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$OpenRegionRemoteCall.call(RSProcedureDispatcher.java:374) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$OpenRegionRemoteCall.call(RSProcedureDispatcher.java:359) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent
[jira] [Commented] (HBASE-20426) Give up replicating anything in S state
[
https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462011#comment-16462011
]
Zheng Hu commented on HBASE-20426:
--
The RB is too slow, so leave a comment here:
{code}
@@ -851,9 +910,32 @@ public class ReplicationSourceManager implements
ReplicationListener {
peer = replicationPeers.getPeer(src.getPeerId());
if (peer == null || !isOldPeer(src.getPeerId(), peer)) {
src.terminate("Recovered queue doesn't belong to any current
peer");
- removeRecoveredSource(src);
+ deleteQueue(queueId);
continue;
}
+// Do not setup recovered queue if a sync replication peer is in
standby state
+if (peer.getPeerConfig().isSyncReplication()) {
+ Pair
stateAndNewState =
+peer.getSyncReplicationStateAndNewState();
+ if
(stateAndNewState.getFirst().equals(SyncReplicationState.STANDBY) ||
+
stateAndNewState.getSecond().equals(SyncReplicationState.STANDBY)) {
+src.terminate("Sync replication peer is in STANDBY state");
+deleteQueue(queueId);
+continue;
+ }
+}
{code}
Why do we need to terminate the recovered source when in S state in
NodeFailoverWorker ? If the cluster is in S state, and is transiting to DA
state, one RS crashed when replaying the remote WALs, I don't think we can
just abandon the WALs from crashed RS, because we need those WALs to replicate
back to another cluster ...
> Give up replicating anything in S state
> ---
>
> Key: HBASE-20426
> URL: https://issues.apache.org/jira/browse/HBASE-20426
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: HBASE-19064
>
> Attachments: HBASE-20426-HBASE-19064-v1.patch,
> HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch,
> HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch,
> HBASE-20426-HBASE-19064-v2.patch, HBASE-20426-HBASE-19064-v3.patch,
> HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch,
> HBASE-20426-HBASE-19064.patch, HBASE-20426-UT.patch
>
>
> When we transit the remote S cluster to DA, and then transit the old A
> cluster to S, it is possible that we still have some entries which have not
> been replicated yet for the old A cluster, and then the async replication
> will be blocked.
> And this may also lead to data inconsistency after we transit it to DA back
> later as these entries will be replicated again, but the new data which are
> replicated from the remote cluster will not be replicated back, which
> introduce a whole in the replication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20426) Give up replicating anything in S state
[ https://issues.apache.org/jira/browse/HBASE-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462013#comment-16462013 ] Duo Zhang commented on HBASE-20426: --- Yes, you're right. If we are trying to transit to DA from S, then we should not abandon the WALs. Let me fix. > Give up replicating anything in S state > --- > > Key: HBASE-20426 > URL: https://issues.apache.org/jira/browse/HBASE-20426 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v1.patch, HBASE-20426-HBASE-19064-v1.patch, > HBASE-20426-HBASE-19064-v2.patch, HBASE-20426-HBASE-19064-v3.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-HBASE-19064.patch, > HBASE-20426-HBASE-19064.patch, HBASE-20426-UT.patch > > > When we transit the remote S cluster to DA, and then transit the old A > cluster to S, it is possible that we still have some entries which have not > been replicated yet for the old A cluster, and then the async replication > will be blocked. > And this may also lead to data inconsistency after we transit it to DA back > later as these entries will be replicated again, but the new data which are > replicated from the remote cluster will not be replicated back, which > introduce a whole in the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[ https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462019#comment-16462019 ] Xiaolin Ha commented on HBASE-19997: Definitely, rolling update RS first and then Master will be work. I think Master.proto of 0.98 and 2.0 is the same. So firstly rolling update regionservers will be more safety. And I think there is no special need to rolling update Master before RS. > [rolling upgrade] 1.x => 2.x > > > Key: HBASE-19997 > URL: https://issues.apache.org/jira/browse/HBASE-19997 > Project: HBase > Issue Type: Umbrella >Reporter: stack >Priority: Blocker > Fix For: 2.1.0 > > Attachments: Screenshot from 2018-05-03 14-43-46.png > > > An umbrella issue of issues needed so folks can do a rolling upgrade from > hbase-1.x to hbase-2.x. > (Recent) Notables: > * hbase-1.x can't read hbase-2.x WALs -- hbase-1.x doesn't know the > AsyncProtobufLogWriter class used writing the WAL -- see > https://issues.apache.org/jira/browse/HBASE-19166?focusedCommentId=16362897&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16362897 > for exception. > ** Might be ok... means WAL split fails on an hbase1 RS... must wait till an > hbase-2.x RS picks up the WAL for it to be split. > * hbase-1 can't open regions from tables created by hbase-2; it can't find > the Table descriptor. See > https://issues.apache.org/jira/browse/HBASE-19116?focusedCommentId=16363276&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16363276 > ** This might be ok if the tables we are doing rolling upgrade over were > written with hbase-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
