[jira] [Commented] (HBASE-21588) Procedure v2 wal splitting implementation
[ https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734786#comment-16734786 ] Jingyun Tian commented on HBASE-21588: -- Seems there are still some problem. Let me try to figure it out > Procedure v2 wal splitting implementation > - > > Key: HBASE-21588 > URL: https://issues.apache.org/jira/browse/HBASE-21588 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21588.master.003.patch, > HBASE-21588.master.004.patch, HBASE-21588.master.005.patch, > HBASE-21588.master.006.patch, HBASE-21588.master.007.patch, > HBASE-21588.master.008.patch, HBASE-21588.master.009.patch, > HBASE-21588.master.010.patch, HBASE-21588.master.011.patch, > HBASE-21588.master.012.patch, HBASE-21588.master.013.patch, > HBASE-21588.master.014.patch, HBASE-21588.master.015.patch, > HBASE-21588.master.016.patch, HBASE-21588.master.017.patch, > HBASE-21588.master.018.patch, HBASE-21588.master.018.patch, > HBASE-21588.master.019.patch, HBASE-21588.master.019.patch > > > create a sub task to submit the implementation of procedure v2 wal splitting -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21538: -- Component/s: read replicas > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task > Components: read replicas >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-21512 > > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21538: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HBASE-21512 Status: Resolved (was: Patch Available) Rebased and pushed to branch HBASE-21512, after fixing the checkstyle issues. Thanks all for reviewing. > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-21512 > > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21657) PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% scan case.
[ https://issues.apache.org/jira/browse/HBASE-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734744#comment-16734744 ] stack commented on HBASE-21657: --- bq. If necessary, I can provide the FlameGraph for HDD too. Nah. Would have been good to confirm that it was same profile but we can guess they were probably the same and CPU the bottleneck. On inlining, you can enable flag on jvm to see jit in action making ruling on whether methods are inlined or not. 6.1 here, https://docs.google.com/document/d/1vZ_k6_pNR1eQxID5u1xFihuPC7FkPaJQW8c4M5eA2AQ/edit#heading=h.izcvb7v6f7pf, has some notes and late in the doc., it describes how to get jit-watch working too which does nice graphics on whats inline vs what is not and if you enable decompile, you can see actual code generated which can help debugging why the time is spent FYI. If the layer above does not completely overlap the level below, I read it as dispatching costs, jit trying to figure what to call when it not straight-forward; i.e. private or statics as you write above; e.g. the step from StoreScanner#next up to the various instances of PrivateCellUtil#estimatedSerializedSizeOf in the first svgs Your patch made for a 40% diff in throughput? It looks like it made about 5% diff in how much cpu the scan takes in overall CPU usage. The simplified dispatch could be the diff. It looks like estimatedSerializedSizeOf is just expensive. There is no reuse of calculated lengths as BBKVComparator#compare does... That could get some cycles back. Caching the serialized size or trying to figure if we have to do size so often could help. bq. avoiding the frequent list extension I see this in the first svg but not in the second. Perhaps you had set initial size to 1000 in second run. We should do this yeah but don't we have the number somewhere of how many rows to return per Scan call so we don't have to guess 1000? bq. Currently, the Cell interface seems try to expose few concept to the upstream user Yes. Was trying to keep it minimal. bq. So i'm not sure whether moving the getSerilizedCell (tags or not ) method into Cell interface is corrent or not even if we gain 40% speedup Getting serialized size is generic enough that it could go up into Cell (good to ask Anoop too). Exposing a version that takes tag shouldn't go into Cell (All current tag API in Cell is deprecated). Regards patched 0.98, we have enough to work with given your current flame graph from 2.0.x. I was just asking. Can help seeing where hbase1 was spending its cpu compared to hbase2. > PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% > scan case. > > > Key: HBASE-21657 > URL: https://issues.apache.org/jira/browse/HBASE-21657 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21657.v1.patch, HBASE-21657.v2.patch, > HBase2.0.4-with-patch.v2.png, HBase2.0.4-without-patch-v2.png, > hbase2.0.4-ssd-scan-traces.2.svg, hbase2.0.4-ssd-scan-traces.svg, > hbase20-ssd-100-scan-traces.svg > > > We are evaluating the performance of branch-2, and find that the throughput > of scan in SSD cluster is almost the same as HDD cluster. so I made a > FlameGraph on RS, and found that the > PrivateCellUtil#estimatedSerializedSizeOf cost about 29% cpu, Obviously, it > has been the bottleneck in 100% scan case. > See the [^hbase20-ssd-100-scan-traces.svg] > BTW, in our XiaoMi branch, we introduce a > HRegion#updateReadRequestsByCapacityUnitPerSecond to sum up the size of cells > (for metric monitor), so it seems the performance loss was amplified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21588) Procedure v2 wal splitting implementation
[
https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734724#comment-16734724
]
Hadoop QA commented on HBASE-21588:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
6s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch passed checkstyle in hbase-protocol-shaded
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} The patch passed checkstyle in hbase-common {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} hbase-server: The patch generated 0 new + 271
unchanged - 1 fixed = 271 total (was 272) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 35s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 22s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 0s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Commented] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[
https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734688#comment-16734688
]
Tak Lon (Stephen) Wu commented on HBASE-21457:
--
The addendum looks good to me, although I went back and searched the master
branch and find more places needed to be addressed with WAL filesystem issue
(I'm mot sure if we'd like to address them in this JIRA).
Must fix with {{CommonFSUtils.getWALRootDir}} for *WAL root directory* with the
correct filesystem:
*
[org/apache/hadoop/hbase/util/HBaseFsck.java/HBaseFsck.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java#L1682-L1684]
*
[org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALEntryStream.java#L315]
The following test classes also need some refactoring to use {{walFs}} or/and
{{CommonFSUtils.getWALRootDir}} to make it clear to operator to use WAL
filesystem, but it's fine because the setup was using the root file system for
both root directory and WAL directory.
*
[org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java#testSyncRunnerIndexOverflow|https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java#L105]
*
[org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java#testUnflushedSeqIdTracking|https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java#L148]
*
[org/apache/hadoop/hbase/fs/TestBlockReorderMultiBlocks.java|https://github.com/apache/hbase/blob/6dd5383033c5b0f84bd1de081a7fdadabe1658b8/hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorderMultiBlocks.java#L144]
*
[org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner|https://github.com/apache/hbase/blob/master/hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner.java#L154]
> BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
> ---
>
> Key: HBASE-21457
> URL: https://issues.apache.org/jira/browse/HBASE-21457
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Janos Gub
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt,
> 21457.v4.txt, HBASE-21457.add.patch
>
>
> Janos reported seeing backup test failure when testing a local HDFS for WALs
> while using WASB/ADLS only for store files.
> Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase
> root dir for retrieving WAL files.
> We should use the helper methods from CommonFSUtils.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15728: -- Release Note: Adds below flush, split, and compaction metrics + // split related metrics + private MutableFastCounter splitRequest; + private MutableFastCounter splitSuccess; + private MetricHistogram splitTimeHisto; + + // flush related metrics + private MetricHistogram flushTimeHisto; + private MetricHistogram flushMemstoreSizeHisto; + private MetricHistogram flushOutputSizeHisto; + private MutableFastCounter flushedMemstoreBytes; + private MutableFastCounter flushedOutputBytes; + + // compaction related metrics + private MetricHistogram compactionTimeHisto; + private MetricHistogram compactionInputFileCountHisto; + private MetricHistogram compactionInputSizeHisto; + private MetricHistogram compactionOutputFileCountHisto; + private MetricHistogram compactionOutputSizeHisto; + private MutableFastCounter compactedInputBytes; + private MutableFastCounter compactedOutputBytes; + + private MetricHistogram majorCompactionTimeHisto; + private MetricHistogram majorCompactionInputFileCountHisto; + private MetricHistogram majorCompactionInputSizeHisto; + private MetricHistogram majorCompactionOutputFileCountHisto; + private MetricHistogram majorCompactionOutputSizeHisto; + private MutableFastCounter majorCompactedInputBytes; + private MutableFastCounter majorCompactedOutputBytes; > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > HBASE-15728.branch-2.addendum.001.patch, HBASE-15728.master.001.patch, > HBASE-15728.master.addendum.001.patch, hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 11:09 PM: - Changes that won't require a backport. Making a note of them now for handling later. Needs release note: * HBASE-16499 slow replication for small HBase clusters * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor Does not: * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. Needs release note: * HBASE-16499 slow replication for small HBase clusters * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor Does not: * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 11:09 PM: - Changes that won't require a backport. Making a note of them now for handling later. Needs release note (new config): * HBASE-16499 slow replication for small HBase clusters * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor Does not: * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. Needs release note: * HBASE-16499 slow replication for small HBase clusters * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor Does not: * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 10:57 PM: - Changes that won't require a backport. Making a note of them now for handling later. Needs release note: * HBASE-16499 slow replication for small HBase clusters * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor Does not: * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21680: --- Summary: Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1 (was: Port HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1) > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734649#comment-16734649 ] Vladimir Rodionov commented on HBASE-21457: --- Addendum fixes WAL FS issues (code does not work if WAL is on a different FS) in IncrementalBackupManager. [~elserj], can you take a look? > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt, HBASE-21457.add.patch > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-21457: -- Attachment: HBASE-21457.add.patch > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt, HBASE-21457.add.patch > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-21677. Resolution: Won't Fix > Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to > branch-1 > > > Key: HBASE-21677 > URL: https://issues.apache.org/jira/browse/HBASE-21677 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 10:47 PM: - Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-19852 HBase Thrift should use a SPNEGO HTTP/hostname principal for checking HTTP Kerberos authentication * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21680) Port HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
Andrew Purtell created HBASE-21680: -- Summary: Port HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1 Key: HBASE-21680 URL: https://issues.apache.org/jira/browse/HBASE-21680 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21676) Port HBASE-21560 (Return a new TableDescriptor for MasterObserver#preModifyTable to allow coprocessor modify the TableDescriptor) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21676: -- Assignee: (was: Andrew Purtell) > Port HBASE-21560 (Return a new TableDescriptor for > MasterObserver#preModifyTable to allow coprocessor modify the > TableDescriptor) to branch-1 > - > > Key: HBASE-21676 > URL: https://issues.apache.org/jira/browse/HBASE-21676 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21677: --- Fix Version/s: (was: 1.5.0) > Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to > branch-1 > > > Key: HBASE-21677 > URL: https://issues.apache.org/jira/browse/HBASE-21677 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734637#comment-16734637 ] Andrew Purtell commented on HBASE-21677: I'm going to close this as Wont Fix because I debated this and decided against it, but want to keep a placeholder. > Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to > branch-1 > > > Key: HBASE-21677 > URL: https://issues.apache.org/jira/browse/HBASE-21677 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21677: -- Assignee: (was: Andrew Purtell) > Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to > branch-1 > > > Key: HBASE-21677 > URL: https://issues.apache.org/jira/browse/HBASE-21677 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734631#comment-16734631 ] Andrew Purtell commented on HBASE-21673: Interesting operability improvements but maybe more than I want to try and deal with: * HBASE-20886 [Auth] Support keytab login in hbase client > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21679) Port HBASE-6028 (Start/Stop compactions at region server level) to branch-1
Andrew Purtell created HBASE-21679: -- Summary: Port HBASE-6028 (Start/Stop compactions at region server level) to branch-1 Key: HBASE-21679 URL: https://issues.apache.org/jira/browse/HBASE-21679 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 10:29 PM: - Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20928 Rewrite calculation of midpoint in binarySearch functions to prevent overflow * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21677: --- Fix Version/s: 1.5.0 > Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to > branch-1 > > > Key: HBASE-21677 > URL: https://issues.apache.org/jira/browse/HBASE-21677 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21588) Procedure v2 wal splitting implementation
[ https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21588: -- Attachment: HBASE-21588.master.019.patch > Procedure v2 wal splitting implementation > - > > Key: HBASE-21588 > URL: https://issues.apache.org/jira/browse/HBASE-21588 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21588.master.003.patch, > HBASE-21588.master.004.patch, HBASE-21588.master.005.patch, > HBASE-21588.master.006.patch, HBASE-21588.master.007.patch, > HBASE-21588.master.008.patch, HBASE-21588.master.009.patch, > HBASE-21588.master.010.patch, HBASE-21588.master.011.patch, > HBASE-21588.master.012.patch, HBASE-21588.master.013.patch, > HBASE-21588.master.014.patch, HBASE-21588.master.015.patch, > HBASE-21588.master.016.patch, HBASE-21588.master.017.patch, > HBASE-21588.master.018.patch, HBASE-21588.master.018.patch, > HBASE-21588.master.019.patch, HBASE-21588.master.019.patch > > > create a sub task to submit the implementation of procedure v2 wal splitting -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21678) Port HBASE-20636 (Introduce two bloom filter type ROWPREFIX and ROWPREFIX_DELIMITED) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21678: --- Fix Version/s: 1.5.0 > Port HBASE-20636 (Introduce two bloom filter type ROWPREFIX and > ROWPREFIX_DELIMITED) to branch-1 > > > Key: HBASE-21678 > URL: https://issues.apache.org/jira/browse/HBASE-21678 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21675) Port HBASE-21642 (CopyTable by reading snapshot and bulkloading will save a lot of time) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21675: --- Fix Version/s: 1.5.0 > Port HBASE-21642 (CopyTable by reading snapshot and bulkloading will save a > lot of time) to branch-1 > > > Key: HBASE-21675 > URL: https://issues.apache.org/jira/browse/HBASE-21675 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21674: --- Fix Version/s: 1.5.0 > Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from > thrift1 server) to branch-1 > > > Key: HBASE-21674 > URL: https://issues.apache.org/jira/browse/HBASE-21674 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 10:20 PM: - Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-20986 Separate the config of block size when we do log splitting and write Hlog * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21476) Support for nanosecond timestamps
[
https://issues.apache.org/jira/browse/HBASE-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734626#comment-16734626
]
Hadoop QA commented on HBASE-21476:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
44s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
21s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} hbase-common: The patch generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} hbase-client: The patch generated 0 new + 69
unchanged - 5 fixed = 69 total (was 74) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
19s{color} | {color:red} hbase-server: The patch generated 1 new + 824
unchanged - 25 fixed = 825 total (was 849) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch passed checkstyle in hbase-shell {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
8s{color} | {color:red} The patch generated 7 new + 407 unchanged - 4 fixed =
414 total (was 411) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 3s{color} | {color:orange} The patch generated 11 new + 747 unchanged - 0
fixed = 758 total (was 747) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 19s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (HBASE-21668) SCM fetch times out for nightlies
[
https://issues.apache.org/jira/browse/HBASE-21668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734621#comment-16734621
]
stack commented on HBASE-21668:
---
Switch to github seems like good idea. +1.
> SCM fetch times out for nightlies
> -
>
> Key: HBASE-21668
> URL: https://issues.apache.org/jira/browse/HBASE-21668
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Attachments: HBASE-21668.master.001.patch
>
>
> Many of the nightly builds are failing with timeout on checkout stage with
> git fetch command. The default timeout is 2 minutes. We could either increase
> the timeout or use shallow clone that would also reduce the used space.
> Error from console output:
> {noformat}
> 08:48:10 Cloning the remote Git repository
> 08:48:10 Cloning with configured refspecs honoured and without tags
> Cloning repository https://git.apache.org/hbase.git
> > git init
> /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@2/component
> # timeout=10
> Fetching upstream changes from https://git.apache.org/hbase.git
> > git --version # timeout=10
> > git fetch --no-tags --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*
> 08:50:15 ERROR: Error cloning remote repo 'origin'
> 08:50:15 hudson.plugins.git.GitException: Command "git fetch --no-tags
> --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*" returned status code 128:
> 08:50:15 stdout:
> 08:50:15 stderr: fatal: unable to access 'https://git.apache.org/hbase.git/':
> Failed to connect to git.apache.org port 443: Connection timed out{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21677: -- Assignee: Andrew Purtell > Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to > branch-1 > > > Key: HBASE-21677 > URL: https://issues.apache.org/jira/browse/HBASE-21677 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15728) Add remaining per-table region / store / flush / compaction related metrics
[ https://issues.apache.org/jira/browse/HBASE-15728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15728: --- Fix Version/s: (was: 1.5.0) > Add remaining per-table region / store / flush / compaction related metrics > > > Key: HBASE-15728 > URL: https://issues.apache.org/jira/browse/HBASE-15728 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Enis Soztutar >Assignee: Xu Cang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-15728.branch-1.001.patch, > HBASE-15728.branch-1.001.patch, HBASE-15728.branch-1.002.patch, > HBASE-15728.branch-2.addendum.001.patch, HBASE-15728.master.001.patch, > HBASE-15728.master.addendum.001.patch, hbase-15728_v1.patch > > > Continuing on the work for per-table metrics, HBASE-15518 and HBASE-15671. > We need to add some remaining metrics at the per-table level, so that we will > have the same metrics reported at the per-regionserver, per-region and > per-table levels. > After this patch, most of the metrics at the RS and all of the per-region > level are also reported at the per-table level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-21457: --- Opened for addendum. > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21678) Port HBASE-20636 (Introduce two bloom filter type ROWPREFIX and ROWPREFIX_DELIMITED) to branch-1
Andrew Purtell created HBASE-21678: -- Summary: Port HBASE-20636 (Introduce two bloom filter type ROWPREFIX and ROWPREFIX_DELIMITED) to branch-1 Key: HBASE-21678 URL: https://issues.apache.org/jira/browse/HBASE-21678 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 9:11 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 9:17 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21208 Bytes#toShort doesn't work without unsafe * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 9:15 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-21164 reportForDuty should do backoff rather than retry * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-20716: Changes the bytes[] conversion done in Bytes and ByteBufferUtils. Instead of doing check unsafe_aligned available everytime, choose the best converter at startup * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 9:07 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21356 bulkLoadHFile API should ensure that rs has the source hfile's write permission * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 9:05 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21372 Set hbase.assignment.maximum.attempts to Long.MAX * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21675) Port HBASE-21642 (CopyTable by reading snapshot and bulkloading will save a lot of time) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21675: -- Assignee: Andrew Purtell > Port HBASE-21642 (CopyTable by reading snapshot and bulkloading will save a > lot of time) to branch-1 > > > Key: HBASE-21675 > URL: https://issues.apache.org/jira/browse/HBASE-21675 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 9:03 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21325 Force to terminate regionserver when abort hang in somewhere * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21677) Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1
Andrew Purtell created HBASE-21677: -- Summary: Port (HBASE-21255 [acl] Refactor TablePermission into three classes) to branch-1 Key: HBASE-21677 URL: https://issues.apache.org/jira/browse/HBASE-21677 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell edited comment on HBASE-21673 at 1/4/19 8:58 PM: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-18735 Provide an option to kill a MiniHBaseCluster without waiting on shutdown * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor was (Author: apurtell): Changes that won't require a backport. Making a note of them now for handling later. * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21676) Port HBASE-21560 (Return a new TableDescriptor for MasterObserver#preModifyTable to allow coprocessor modify the TableDescriptor) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21676: -- Assignee: Andrew Purtell > Port HBASE-21560 (Return a new TableDescriptor for > MasterObserver#preModifyTable to allow coprocessor modify the > TableDescriptor) to branch-1 > - > > Key: HBASE-21676 > URL: https://issues.apache.org/jira/browse/HBASE-21676 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21676) Port HBASE-21560 (Return a new TableDescriptor for MasterObserver#preModifyTable to allow coprocessor modify the TableDescriptor) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734560#comment-16734560 ] Andrew Purtell commented on HBASE-21676: I'm going to close this as Wont Fix because I debated this and decided against it, but want to keep a placeholder. > Port HBASE-21560 (Return a new TableDescriptor for > MasterObserver#preModifyTable to allow coprocessor modify the > TableDescriptor) to branch-1 > - > > Key: HBASE-21676 > URL: https://issues.apache.org/jira/browse/HBASE-21676 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21676) Port HBASE-21560 (Return a new TableDescriptor for MasterObserver#preModifyTable to allow coprocessor modify the TableDescriptor) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-21676. Resolution: Won't Fix > Port HBASE-21560 (Return a new TableDescriptor for > MasterObserver#preModifyTable to allow coprocessor modify the > TableDescriptor) to branch-1 > - > > Key: HBASE-21676 > URL: https://issues.apache.org/jira/browse/HBASE-21676 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21676) Port HBASE-21560 (Return a new TableDescriptor for MasterObserver#preModifyTable to allow coprocessor modify the TableDescriptor) to branch-1
Andrew Purtell created HBASE-21676: -- Summary: Port HBASE-21560 (Return a new TableDescriptor for MasterObserver#preModifyTable to allow coprocessor modify the TableDescriptor) to branch-1 Key: HBASE-21676 URL: https://issues.apache.org/jira/browse/HBASE-21676 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5.0)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21673: --- Description: Umbrella for backports to branch-1 of suitable changes for a new minor (1.5.0). See subtasks. (was: Umbrella for backports to branch-1 of suitable changes for a new minor (1.5.0). At the moment there is only HBASE-21616. See subtasks.) > Final sweep of branch-2+ commits for new branch-1 minor (1.5.0) > --- > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor > (1.5.0). See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21675) Port HBASE-21642 (CopyTable by reading snapshot and bulkloading will save a lot of time) to branch-1
Andrew Purtell created HBASE-21675: -- Summary: Port HBASE-21642 (CopyTable by reading snapshot and bulkloading will save a lot of time) to branch-1 Key: HBASE-21675 URL: https://issues.apache.org/jira/browse/HBASE-21675 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734548#comment-16734548 ] Andrew Purtell commented on HBASE-21674: Include HBASE-21650 > Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from > thrift1 server) to branch-1 > > > Key: HBASE-21674 > URL: https://issues.apache.org/jira/browse/HBASE-21674 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734546#comment-16734546 ] Andrew Purtell commented on HBASE-21673: Changes that won't require a backport. Making a note of them now for handling later. * HBASE-21659 Avoid to load duplicate coprocessors in system config and table descriptor > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
Andrew Purtell created HBASE-21674: -- Summary: Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1 Key: HBASE-21674 URL: https://issues.apache.org/jira/browse/HBASE-21674 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21673: --- Summary: Final sweep of branch-2+ commits for new branch-1 minor (1.5) (was: Final sweep of branch-2+ commits for new branch-1 minor (1.5.0)) > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor > (1.5.0). See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5)
[ https://issues.apache.org/jira/browse/HBASE-21673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21673: --- Description: Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). See subtasks. (was: Umbrella for backports to branch-1 of suitable changes for a new minor (1.5.0). See subtasks.) > Final sweep of branch-2+ commits for new branch-1 minor (1.5) > - > > Key: HBASE-21673 > URL: https://issues.apache.org/jira/browse/HBASE-21673 > Project: HBase > Issue Type: Umbrella >Affects Versions: 1.5.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > > Umbrella for backports to branch-1 of suitable changes for a new minor (1.5). > See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21616) Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21616: --- Issue Type: Sub-task (was: Task) Parent: HBASE-21673 > Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1 > -- > > Key: HBASE-21616 > URL: https://issues.apache.org/jira/browse/HBASE-21616 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > > Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21673) Final sweep of branch-2+ commits for new branch-1 minor (1.5.0)
Andrew Purtell created HBASE-21673: -- Summary: Final sweep of branch-2+ commits for new branch-1 minor (1.5.0) Key: HBASE-21673 URL: https://issues.apache.org/jira/browse/HBASE-21673 Project: HBase Issue Type: Umbrella Affects Versions: 1.5.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Umbrella for backports to branch-1 of suitable changes for a new minor (1.5.0). At the moment there is only HBASE-21616. See subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21616) Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21616: -- Assignee: Andrew Purtell > Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1 > -- > > Key: HBASE-21616 > URL: https://issues.apache.org/jira/browse/HBASE-21616 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > > Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21476) Support for nanosecond timestamps
[ https://issues.apache.org/jira/browse/HBASE-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734528#comment-16734528 ] Sean Busbey commented on HBASE-21476: - If you modify our yetus personality's definition for "personality_modules" ([line 179 currently defines arguments we pass for unit tests|https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L179]) to include that flag then you can get a test run with the flag active. We'll need a plan on how testing this will work going forward, because we a) don't want to run all the tests with just this flag set, b) probably we don't want to run all the tests with it on and off in precommit, c) maybe we want both for nightly tests? One way to do this is to define a profile that has a selection of tests that provide good coverage of this feature as well as the needed flag to have them run with the table attribute set. Then we can add to our yetus personality a test that expressly is running unit tests with that profile active. These are things that should be included in the scope document, probably by expanding the section on testing. > Support for nanosecond timestamps > - > > Key: HBASE-21476 > URL: https://issues.apache.org/jira/browse/HBASE-21476 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.1.1 >Reporter: Andrey Elenskiy >Assignee: Andrey Elenskiy >Priority: Major > Labels: features, patch > Attachments: Apache HBase - Nanosecond Timestamps v1.pdf, > HBASE-21476.branch-2.1.0003.patch, HBASE-21476.branch-2.1.0004.patch, > nanosecond_timestamps_v1.patch, nanosecond_timestamps_v2.patch > > > Introducing a new table attribute "NANOSECOND_TIMESTAMPS" to tell HBase to > handle timestamps with nanosecond precision. This is useful for applications > that timestamp updates at the source with nanoseconds and still want features > like column family TTL and "hbase.hstore.time.to.purge.deletes" to work. > The attribute should be specified either on new tables or on existing tables > which have timestamps only with nanosecond precision. There's no migration > from milliseconds to nanoseconds for already existing tables. We could add > this migration as part of compaction if you think that would be useful, but > that would obviously make the change more complex. > I've added a new EnvironmentEdge method "currentTimeNano()" that uses > [java.time.Instant|https://docs.oracle.com/javase/8/docs/api/java/time/Instant.html] > to get time in nanoseconds which means it will only work with Java 8. The > idea is to gradually replace all places where "EnvironmentEdge.currentTime()" > is used to have HBase working purely with nanoseconds (which is a > prerequisite for HBASE-14070). Also, I've refactored ScanInfo and > PartitionedMobCompactor to expect TableDescriptor as an argument which makes > code a little cleaner and easier to extend. > Couple more points: > - column family TTL (specified in seconds) and > "hbase.hstore.time.to.purge.deletes" (specified in milliseconds) options > don't need to be changed, those are adjusted automatically. > - Per cell TTL needs to be scaled by clients accordingly after > "NANOSECOND_TIMESTAMPS" table attribute is specified. > Looking for everyone's feedback to know if that's a worthwhile direction. > Will add more comprehensive tests in a later patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21362) Disable printing of stack-trace in shell when quotas are violated
[
https://issues.apache.org/jira/browse/HBASE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734478#comment-16734478
]
Josh Elser commented on HBASE-21362:
LGTM. Thanks for the nice little improvement, [~jatsakthi].
> Disable printing of stack-trace in shell when quotas are violated
> -
>
> Key: HBASE-21362
> URL: https://issues.apache.org/jira/browse/HBASE-21362
> Project: HBase
> Issue Type: Sub-task
> Components: shell
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-21362.master.001.patch,
> hbase-21362.master.002.patch, hbase-21362.master.003.patch
>
>
> When quotas are violated, a 'Quota violated on #table. Due to
> #ViolationPolicy, #Action is not allowed' message should suffice. The current
> trace in shell looks like this:
> {noformat}
> hbase(main):009:0> put 't2','r1','cf1:c','val'
> ERROR: org.apache.hadoop.hbase.quotas.SpaceLimitingException: NO_WRITES Puts
> are disallowed due to a space quota.
> at
> org.apache.hadoop.hbase.quotas.policies.NoWritesViolationPolicyEnforcement.check(NoWritesViolationPolicyEnforcement.java:46)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2779)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42000)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21476) Support for nanosecond timestamps
[ https://issues.apache.org/jira/browse/HBASE-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Elenskiy updated HBASE-21476: Attachment: HBASE-21476.branch-2.1.0004.patch > Support for nanosecond timestamps > - > > Key: HBASE-21476 > URL: https://issues.apache.org/jira/browse/HBASE-21476 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.1.1 >Reporter: Andrey Elenskiy >Assignee: Andrey Elenskiy >Priority: Major > Labels: features, patch > Attachments: Apache HBase - Nanosecond Timestamps v1.pdf, > HBASE-21476.branch-2.1.0003.patch, HBASE-21476.branch-2.1.0004.patch, > nanosecond_timestamps_v1.patch, nanosecond_timestamps_v2.patch > > > Introducing a new table attribute "NANOSECOND_TIMESTAMPS" to tell HBase to > handle timestamps with nanosecond precision. This is useful for applications > that timestamp updates at the source with nanoseconds and still want features > like column family TTL and "hbase.hstore.time.to.purge.deletes" to work. > The attribute should be specified either on new tables or on existing tables > which have timestamps only with nanosecond precision. There's no migration > from milliseconds to nanoseconds for already existing tables. We could add > this migration as part of compaction if you think that would be useful, but > that would obviously make the change more complex. > I've added a new EnvironmentEdge method "currentTimeNano()" that uses > [java.time.Instant|https://docs.oracle.com/javase/8/docs/api/java/time/Instant.html] > to get time in nanoseconds which means it will only work with Java 8. The > idea is to gradually replace all places where "EnvironmentEdge.currentTime()" > is used to have HBase working purely with nanoseconds (which is a > prerequisite for HBASE-14070). Also, I've refactored ScanInfo and > PartitionedMobCompactor to expect TableDescriptor as an argument which makes > code a little cleaner and easier to extend. > Couple more points: > - column family TTL (specified in seconds) and > "hbase.hstore.time.to.purge.deletes" (specified in milliseconds) options > don't need to be changed, those are adjusted automatically. > - Per cell TTL needs to be scaled by clients accordingly after > "NANOSECOND_TIMESTAMPS" table attribute is specified. > Looking for everyone's feedback to know if that's a worthwhile direction. > Will add more comprehensive tests in a later patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[
https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734319#comment-16734319
]
Hudson commented on HBASE-21512:
Results for branch HBASE-21512
[build #39 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/39/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/39//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/39//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/39//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/39//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
> --
>
> Key: HBASE-21512
> URL: https://issues.apache.org/jira/browse/HBASE-21512
> Project: HBase
> Issue Type: Umbrella
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> At least for the RSProcedureDispatcher, with CompletableFuture we do not need
> to set a delay and use a thread pool any more, which could reduce the
> resource usage and also the latency.
> Once this is done, I think we can remove the ClusterConnection completely,
> and start to rewrite the old sync client based on the async client, which
> could reduce the code base a lot for our client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21668) SCM fetch times out for nightlies
[
https://issues.apache.org/jira/browse/HBASE-21668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734299#comment-16734299
]
Sean Busbey commented on HBASE-21668:
-
forgot to add, we can check some runs after this patch lands to see if I'm
remembering the Yetus behavior correctly. If so we can figure out if it's worth
an improvement to limit how much yetus pulls down.
> SCM fetch times out for nightlies
> -
>
> Key: HBASE-21668
> URL: https://issues.apache.org/jira/browse/HBASE-21668
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Attachments: HBASE-21668.master.001.patch
>
>
> Many of the nightly builds are failing with timeout on checkout stage with
> git fetch command. The default timeout is 2 minutes. We could either increase
> the timeout or use shallow clone that would also reduce the used space.
> Error from console output:
> {noformat}
> 08:48:10 Cloning the remote Git repository
> 08:48:10 Cloning with configured refspecs honoured and without tags
> Cloning repository https://git.apache.org/hbase.git
> > git init
> /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@2/component
> # timeout=10
> Fetching upstream changes from https://git.apache.org/hbase.git
> > git --version # timeout=10
> > git fetch --no-tags --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*
> 08:50:15 ERROR: Error cloning remote repo 'origin'
> 08:50:15 hudson.plugins.git.GitException: Command "git fetch --no-tags
> --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*" returned status code 128:
> 08:50:15 stdout:
> 08:50:15 stderr: fatal: unable to access 'https://git.apache.org/hbase.git/':
> Failed to connect to git.apache.org port 443: Connection timed out{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21668) SCM fetch times out for nightlies
[
https://issues.apache.org/jira/browse/HBASE-21668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734298#comment-16734298
]
Sean Busbey commented on HBASE-21668:
-
That's a good improvement. How about we switch to shallow through the patch
here and we edit the Jenkins job to start consuming from github?
Worst case, a github outage means we miss a nightly test. doesn't seem too bad.
If that works for everyone, I'm +1 on the current patch. I can edit the Jenkins
job unless someone beats me to it.
> SCM fetch times out for nightlies
> -
>
> Key: HBASE-21668
> URL: https://issues.apache.org/jira/browse/HBASE-21668
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Attachments: HBASE-21668.master.001.patch
>
>
> Many of the nightly builds are failing with timeout on checkout stage with
> git fetch command. The default timeout is 2 minutes. We could either increase
> the timeout or use shallow clone that would also reduce the used space.
> Error from console output:
> {noformat}
> 08:48:10 Cloning the remote Git repository
> 08:48:10 Cloning with configured refspecs honoured and without tags
> Cloning repository https://git.apache.org/hbase.git
> > git init
> /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@2/component
> # timeout=10
> Fetching upstream changes from https://git.apache.org/hbase.git
> > git --version # timeout=10
> > git fetch --no-tags --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*
> 08:50:15 ERROR: Error cloning remote repo 'origin'
> 08:50:15 hudson.plugins.git.GitException: Command "git fetch --no-tags
> --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*" returned status code 128:
> 08:50:15 stdout:
> 08:50:15 stderr: fatal: unable to access 'https://git.apache.org/hbase.git/':
> Failed to connect to git.apache.org port 443: Connection timed out{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21672) Allow skipping HDFS block distribution computation
[
https://issues.apache.org/jira/browse/HBASE-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734269#comment-16734269
]
Sean Busbey commented on HBASE-21672:
-
this feels a bit hacky. Shouldn't this either be a no-op for filesystems that
don't have locality, or something we can just ask the filesystem? (even if the
current FileSystem API can't do that)
> Allow skipping HDFS block distribution computation
> --
>
> Key: HBASE-21672
> URL: https://issues.apache.org/jira/browse/HBASE-21672
> Project: HBase
> Issue Type: Improvement
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Labels: S3
>
> We should have a configuration to skip HDFS block distribution calculation in
> HBase. For example on file systems that do not surface locality such as S3,
> calculating block distribution would not be any useful.
> Currentlly, we do not have a way to skip hdfs block distribution computation.
> For this, we can provide a new configuration key, say
> {{hbase.block.distribution.skip.computation}} (which would be {{false}} by
> default).
> Users using filesystems such as s3 may choose to make this {{true}}, thus
> skipping block distribution computation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21588) Procedure v2 wal splitting implementation
[
https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734207#comment-16734207
]
Hadoop QA commented on HBASE-21588:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch passed checkstyle in hbase-protocol-shaded
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} The patch passed checkstyle in hbase-common {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} hbase-server: The patch generated 0 new + 271
unchanged - 1 fixed = 271 total (was 272) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
9s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}230m 38s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
3s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}286m 29s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Commented] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[
https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734181#comment-16734181
]
Hudson commented on HBASE-21512:
Results for branch HBASE-21512
[build #40 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/40/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/40//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/40//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/40//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
> --
>
> Key: HBASE-21512
> URL: https://issues.apache.org/jira/browse/HBASE-21512
> Project: HBase
> Issue Type: Umbrella
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> At least for the RSProcedureDispatcher, with CompletableFuture we do not need
> to set a delay and use a thread pool any more, which could reduce the
> resource usage and also the latency.
> Once this is done, I think we can remove the ClusterConnection completely,
> and start to rewrite the old sync client based on the async client, which
> could reduce the code base a lot for our client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21668) SCM fetch times out for nightlies
[
https://issues.apache.org/jira/browse/HBASE-21668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734172#comment-16734172
]
Peter Somogyi commented on HBASE-21668:
---
4:11 for {{git fetch --no-tags https://git.apache.org/hbase.git
+refs/heads/\*:refs/remotes/origin/\*}}
1:26 for {{git fetch --no-tags https://github.com/apache/hbase.git
+refs/heads/\*:refs/remotes/origin/\*}}
> SCM fetch times out for nightlies
> -
>
> Key: HBASE-21668
> URL: https://issues.apache.org/jira/browse/HBASE-21668
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Critical
> Attachments: HBASE-21668.master.001.patch
>
>
> Many of the nightly builds are failing with timeout on checkout stage with
> git fetch command. The default timeout is 2 minutes. We could either increase
> the timeout or use shallow clone that would also reduce the used space.
> Error from console output:
> {noformat}
> 08:48:10 Cloning the remote Git repository
> 08:48:10 Cloning with configured refspecs honoured and without tags
> Cloning repository https://git.apache.org/hbase.git
> > git init
> /home/jenkins/jenkins-slave/workspace/HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@2/component
> # timeout=10
> Fetching upstream changes from https://git.apache.org/hbase.git
> > git --version # timeout=10
> > git fetch --no-tags --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*
> 08:50:15 ERROR: Error cloning remote repo 'origin'
> 08:50:15 hudson.plugins.git.GitException: Command "git fetch --no-tags
> --progress https://git.apache.org/hbase.git
> +refs/heads/*:refs/remotes/origin/*" returned status code 128:
> 08:50:15 stdout:
> 08:50:15 stderr: fatal: unable to access 'https://git.apache.org/hbase.git/':
> Failed to connect to git.apache.org port 443: Connection timed out{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734128#comment-16734128 ] Duo Zhang commented on HBASE-21538: --- We have a RetryCounter in this class so I just reuse it. The old implementation will use the client retry logic and then plus the RetryCounter, I do not think this is necessary... > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734131#comment-16734131 ] Duo Zhang commented on HBASE-21538: --- And most of the changes under HBASE-21512 will be simple, just replace the old blocking call with a future.get(), which usually could keep the old behavior. The intention here is to remove the blocking client implementation, so we do not need to maintain the code for two clients. Can open new issues to rewrite with an complete asynchronous way. > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734129#comment-16734129 ] Peter Somogyi commented on HBASE-21538: --- Thanks for the clarification. +1 > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21630) [shell] Define ENDKEY == STOPROW (we have ENDROW)
[
https://issues.apache.org/jira/browse/HBASE-21630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734122#comment-16734122
]
Hudson commented on HBASE-21630:
Results for branch branch-2.1
[build #741 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/741/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/741//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/741//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/741//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [shell] Define ENDKEY == STOPROW (we have ENDROW)
> -
>
> Key: HBASE-21630
> URL: https://issues.apache.org/jira/browse/HBASE-21630
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Nihal Jain
>Priority: Trivial
> Labels: beginner
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-20630.master.001.patch
>
>
> Small fix... In HTableDescriptor, the end row is labelled ENDKEY. In the
> shell, I should be able to scan to the ENDKEY... but I can't. Scan takes
> STOPROW or ENDROW.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734109#comment-16734109 ] Peter Somogyi commented on HBASE-21538: --- Change looks good to me. I assume the timeouts work the same way from client perspective even though the retry logic is a bit different due to the async call. > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734088#comment-16734088 ] Duo Zhang commented on HBASE-21538: --- The failed UTs are not related. > Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection > --- > > Key: HBASE-21538 > URL: https://issues.apache.org/jira/browse/HBASE-21538 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21538-HBASE-21512-v1.patch, > HBASE-21538-HBASE-21512-v2.patch, HBASE-21538-HBASE-21512-v2.patch, > HBASE-21538-HBASE-21512-v3.patch, HBASE-21538-HBASE-21512.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734067#comment-16734067
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #62 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/62/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/62//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/62//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/62//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21630) [shell] Define ENDKEY == STOPROW (we have ENDROW)
[
https://issues.apache.org/jira/browse/HBASE-21630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734073#comment-16734073
]
Hudson commented on HBASE-21630:
Results for branch branch-2
[build #1592 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1592/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1592//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1592//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1592//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [shell] Define ENDKEY == STOPROW (we have ENDROW)
> -
>
> Key: HBASE-21630
> URL: https://issues.apache.org/jira/browse/HBASE-21630
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Nihal Jain
>Priority: Trivial
> Labels: beginner
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-20630.master.001.patch
>
>
> Small fix... In HTableDescriptor, the end row is labelled ENDKEY. In the
> shell, I should be able to scan to the ENDKEY... but I can't. Scan takes
> STOPROW or ENDROW.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21630) [shell] Define ENDKEY == STOPROW (we have ENDROW)
[
https://issues.apache.org/jira/browse/HBASE-21630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734045#comment-16734045
]
Hudson commented on HBASE-21630:
Results for branch master
[build #696 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/696/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/696//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/696//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/696//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [shell] Define ENDKEY == STOPROW (we have ENDROW)
> -
>
> Key: HBASE-21630
> URL: https://issues.apache.org/jira/browse/HBASE-21630
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Nihal Jain
>Priority: Trivial
> Labels: beginner
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-20630.master.001.patch
>
>
> Small fix... In HTableDescriptor, the end row is labelled ENDKEY. In the
> shell, I should be able to scan to the ENDKEY... but I can't. Scan takes
> STOPROW or ENDROW.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21670) Add 2.0.4 to download page
[
https://issues.apache.org/jira/browse/HBASE-21670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734044#comment-16734044
]
Hudson commented on HBASE-21670:
Results for branch master
[build #696 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/696/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/696//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/696//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/696//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add 2.0.4 to download page
> --
>
> Key: HBASE-21670
> URL: https://issues.apache.org/jira/browse/HBASE-21670
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.4
>
>
> Update the download page on site to have 2.0.4 instead of 2.0.3.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21672) Allow skipping HDFS block distribution computation
[
https://issues.apache.org/jira/browse/HBASE-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21672:
---
Description:
We should have a configuration to skip HDFS block distribution calculation in
HBase. For example on file systems that do not surface locality such as S3,
calculating block distribution would not be any useful.
Currentlly, we do not have a way to skip hdfs block distribution computation.
For this, we can provide a new configuration key, say
{{hbase.block.distribution.skip.computation}} (which would be {{false}} by
default).
Users using filesystems such as s3 may choose to make this {{true}}, thus
skipping block distribution computation.
was:
We should have a configuration to skip HDFS block distribution calculation in
HBase. For example on file systems that do not surface locality such as S3,
calculating block distribution would not be any useful, we should have a way to
skip hdfs block distribution.
For this, we can provide a new configuration key, say
{{hbase.block.distribution.skip.computation}}, which would be {{false}} by
default. Users using filesystems such as s3 may choose to make this {{true}},
thus skipping block distribution computation.
> Allow skipping HDFS block distribution computation
> --
>
> Key: HBASE-21672
> URL: https://issues.apache.org/jira/browse/HBASE-21672
> Project: HBase
> Issue Type: Improvement
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Labels: S3
>
> We should have a configuration to skip HDFS block distribution calculation in
> HBase. For example on file systems that do not surface locality such as S3,
> calculating block distribution would not be any useful.
> Currentlly, we do not have a way to skip hdfs block distribution computation.
> For this, we can provide a new configuration key, say
> {{hbase.block.distribution.skip.computation}} (which would be {{false}} by
> default).
> Users using filesystems such as s3 may choose to make this {{true}}, thus
> skipping block distribution computation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21672) Allow skipping HDFS block distribution computation
Nihal Jain created HBASE-21672:
--
Summary: Allow skipping HDFS block distribution computation
Key: HBASE-21672
URL: https://issues.apache.org/jira/browse/HBASE-21672
Project: HBase
Issue Type: Improvement
Reporter: Nihal Jain
Assignee: Nihal Jain
We should have a configuration to skip HDFS block distribution calculation in
HBase. For example on file systems that do not surface locality such as S3,
calculating block distribution would not be any useful, we should have a way to
skip hdfs block distribution.
For this, we can provide a new configuration key, say
{{hbase.block.distribution.skip.computation}}, which would be {{false}} by
default. Users using filesystems such as s3 may choose to make this {{true}},
thus skipping block distribution computation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21672) Allow skipping HDFS block distribution computation
[
https://issues.apache.org/jira/browse/HBASE-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21672:
---
Labels: S3 (was: )
> Allow skipping HDFS block distribution computation
> --
>
> Key: HBASE-21672
> URL: https://issues.apache.org/jira/browse/HBASE-21672
> Project: HBase
> Issue Type: Improvement
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Labels: S3
>
> We should have a configuration to skip HDFS block distribution calculation in
> HBase. For example on file systems that do not surface locality such as S3,
> calculating block distribution would not be any useful, we should have a way
> to skip hdfs block distribution.
> For this, we can provide a new configuration key, say
> {{hbase.block.distribution.skip.computation}}, which would be {{false}} by
> default. Users using filesystems such as s3 may choose to make this {{true}},
> thus skipping block distribution computation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21538) Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection
[
https://issues.apache.org/jira/browse/HBASE-21538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734012#comment-16734012
]
Hadoop QA commented on HBASE-21538:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-21512 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
1s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
5s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} HBASE-21512 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
5s{color} | {color:red} hbase-server: The patch generated 3 new + 78 unchanged
- 3 fixed = 81 total (was 81) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 37s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
15s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 58s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.cleaner.TestSnapshotFromMaster |
| | hadoop.hbase.replication.TestSyncReplicationStandbyKillRS |
| | hadoop.hbase.replication.TestSyncReplicationActive |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21538 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-21159) Add shell command to switch throttle on or off
[
https://issues.apache.org/jira/browse/HBASE-21159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734006#comment-16734006
]
Hadoop QA commented on HBASE-21159:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 6 new + 149 unchanged - 2 fixed =
155 total (was 151) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 8s{color} | {color:orange} The patch generated 14 new + 270 unchanged - 0
fixed = 284 total (was 270) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}142m
36s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
11s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
26s{color} | {color:green} The patch does not generate ASF License
[jira] [Updated] (HBASE-21588) Procedure v2 wal splitting implementation
[ https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21588: - Attachment: HBASE-21588.master.019.patch > Procedure v2 wal splitting implementation > - > > Key: HBASE-21588 > URL: https://issues.apache.org/jira/browse/HBASE-21588 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21588.master.003.patch, > HBASE-21588.master.004.patch, HBASE-21588.master.005.patch, > HBASE-21588.master.006.patch, HBASE-21588.master.007.patch, > HBASE-21588.master.008.patch, HBASE-21588.master.009.patch, > HBASE-21588.master.010.patch, HBASE-21588.master.011.patch, > HBASE-21588.master.012.patch, HBASE-21588.master.013.patch, > HBASE-21588.master.014.patch, HBASE-21588.master.015.patch, > HBASE-21588.master.016.patch, HBASE-21588.master.017.patch, > HBASE-21588.master.018.patch, HBASE-21588.master.018.patch, > HBASE-21588.master.019.patch > > > create a sub task to submit the implementation of procedure v2 wal splitting -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21588) Procedure v2 wal splitting implementation
[ https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21588: - Attachment: (was: HBASE-21588.master.019.patch) > Procedure v2 wal splitting implementation > - > > Key: HBASE-21588 > URL: https://issues.apache.org/jira/browse/HBASE-21588 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21588.master.003.patch, > HBASE-21588.master.004.patch, HBASE-21588.master.005.patch, > HBASE-21588.master.006.patch, HBASE-21588.master.007.patch, > HBASE-21588.master.008.patch, HBASE-21588.master.009.patch, > HBASE-21588.master.010.patch, HBASE-21588.master.011.patch, > HBASE-21588.master.012.patch, HBASE-21588.master.013.patch, > HBASE-21588.master.014.patch, HBASE-21588.master.015.patch, > HBASE-21588.master.016.patch, HBASE-21588.master.017.patch, > HBASE-21588.master.018.patch, HBASE-21588.master.018.patch > > > create a sub task to submit the implementation of procedure v2 wal splitting -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21588) Procedure v2 wal splitting implementation
[ https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733916#comment-16733916 ] Jingyun Tian edited comment on HBASE-21588 at 1/4/19 9:45 AM: -- [ERROR] org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Time elapsed: 24.063 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.createTable(TestServerCrashProcedure.java:228) at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.testRecoveryAndDoubleExecution(TestServerCrashProcedure.java:138) at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.testCrashTargetRs(TestServerCrashProcedure.java:112) [ERROR] org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Time elapsed: 24.357 s <<< ERROR! According this log, this TestServerCrashProcedure is not stuck but exceeds time limit since it's too large. it costs 10min 52sec on my own desktop. Let me retry this UT with bounded number of split writer which could make it more stable. was (Author: tianjingyun): [ERROR] org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Time elapsed: 24.063 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.createTable(TestServerCrashProcedure.java:228) at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.testRecoveryAndDoubleExecution(TestServerCrashProcedure.java:138) at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.testCrashTargetRs(TestServerCrashProcedure.java:112) [ERROR] org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Time elapsed: 24.357 s <<< ERROR! According this log, this TestServerCrashProcedure is not stuck but exceeds time limit since it's too large. it costs 10min 52sec on my own desktop. Let me split this UT. > Procedure v2 wal splitting implementation > - > > Key: HBASE-21588 > URL: https://issues.apache.org/jira/browse/HBASE-21588 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21588.master.003.patch, > HBASE-21588.master.004.patch, HBASE-21588.master.005.patch, > HBASE-21588.master.006.patch, HBASE-21588.master.007.patch, > HBASE-21588.master.008.patch, HBASE-21588.master.009.patch, > HBASE-21588.master.010.patch, HBASE-21588.master.011.patch, > HBASE-21588.master.012.patch, HBASE-21588.master.013.patch, > HBASE-21588.master.014.patch, HBASE-21588.master.015.patch, > HBASE-21588.master.016.patch, HBASE-21588.master.017.patch, > HBASE-21588.master.018.patch, HBASE-21588.master.018.patch, > HBASE-21588.master.019.patch > > > create a sub task to submit the implementation of procedure v2 wal splitting -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21669) Move branch-2.0 to 2.0.5-SNAPSHOT version after release of 2.0.4
[
https://issues.apache.org/jira/browse/HBASE-21669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733960#comment-16733960
]
Hudson commented on HBASE-21669:
Results for branch branch-2.0
[build #1226 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Move branch-2.0 to 2.0.5-SNAPSHOT version after release of 2.0.4
>
>
> Key: HBASE-21669
> URL: https://issues.apache.org/jira/browse/HBASE-21669
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.5
>
>
> $ mvn clean org.codehaus.mojo:versions-maven-plugin:2.5:set
> -DnewVersion=2.0.5-SNAPSHOT
> $ find . -name pom.xml -exec git add {} \;
> $ git commit ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21630) [shell] Define ENDKEY == STOPROW (we have ENDROW)
[
https://issues.apache.org/jira/browse/HBASE-21630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733961#comment-16733961
]
Hudson commented on HBASE-21630:
Results for branch branch-2.0
[build #1226 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1226//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [shell] Define ENDKEY == STOPROW (we have ENDROW)
> -
>
> Key: HBASE-21630
> URL: https://issues.apache.org/jira/browse/HBASE-21630
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Nihal Jain
>Priority: Trivial
> Labels: beginner
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-20630.master.001.patch
>
>
> Small fix... In HTableDescriptor, the end row is labelled ENDKEY. In the
> shell, I should be able to scan to the ENDKEY... but I can't. Scan takes
> STOPROW or ENDROW.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21657) PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% scan case.
[
https://issues.apache.org/jira/browse/HBASE-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733921#comment-16733921
]
Zheng Hu commented on HBASE-21657:
--
{quote}Whats the hdd flamegraph look like? It was same version? Where is it
spending time? In same place?
{quote}
I've tested both HDD and SSD in the same version, and the throughtput almost
the same (about ~2000 ops/sec for a single node). But I did not catch the
FlameGraph for HDD node before. If necessary, I can provide the FlameGraph for
HDD too.
{quote}Which are these Zheng Hu ? And when you say, did not work, is it that
they are not inlining?
{quote}
I mean the PrivateCellUtil#estimatedSerializedSizeOf method, its implementation
is not so lightweight, use the instanceof and class cast frequently. there's a
post[1] says: _Methods that can be inlined include static, private or final
methods but also public methods if it can be determined that they are not
overridden._
So I guess the hot PrivateCellUtil#estimatedSerializedSizeOf was not inline,
and tried to move the estimatedSerializedSizeOf from ExtendedCell to Cell for
comparison. Seems the instanceof & class cast also cost some cpu, but the
method inline seems still did not work after patch.v2 (because of the kinds of
method overridden ? , I'm not sure... ) we can see that: after applying the
patch.v2, there's still large percent of cpu wasted on
PrivateCellUtil#estimatedSerializedSizeOf.
Another optimization is here, we can set a determinated array length for
*avoiding the frequent list extension* when in a big scan such as the YCSB case
, scan with a limit which is a random length between 1 to 1000.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
index f788a86..6a8b596 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
@@ -3484,7 +3484,7 @@ public class RSRpcServices implements
HBaseRPCErrorHandler,
MutableObject lastBlock = new MutableObject<>();
boolean scannerClosed = false;
try {
- List results = new ArrayList<>();
+ List results = new ArrayList<>(rows);
{code}
{quote}Adding serialized size to Cell Interface is a radical change but having
defaults makes it easier and hard to argue w/ a 40% speedup
{quote}
Currently, the Cell interface seems try to expose few concept to the upstream
user, For example, Tag won't be used by upstream user, but is used by HBase
server side (ACL & Mob feature depends on tags). we removed the tag bytes
before sending the server side cell bytes to user when encoding. So i'm not
sure whether moving the getSerilizedCell (tags or not ) method into Cell
interface is corrent or not even if we gain 40% speedup. How do you think ?
[~anoop.hbase].
{quote}Could you get a flame graph from your current prod to see what its
flamegraph looks like – where branch-1 is spending its time Scanning?
{quote}
Our prod branch is HBase0.98 now , and we backport many features and made much
customization. seems it's not a good contrast. But maybe I can give a
FlameGraph based on the community branch-1.
Thanks.
1. [https://techblug.wordpress.com/2013/08/19/java-jit-compiler-inlining/]
> PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100%
> scan case.
>
>
> Key: HBASE-21657
> URL: https://issues.apache.org/jira/browse/HBASE-21657
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21657.v1.patch, HBASE-21657.v2.patch,
> HBase2.0.4-with-patch.v2.png, HBase2.0.4-without-patch-v2.png,
> hbase2.0.4-ssd-scan-traces.2.svg, hbase2.0.4-ssd-scan-traces.svg,
> hbase20-ssd-100-scan-traces.svg
>
>
> We are evaluating the performance of branch-2, and find that the throughput
> of scan in SSD cluster is almost the same as HDD cluster. so I made a
> FlameGraph on RS, and found that the
> PrivateCellUtil#estimatedSerializedSizeOf cost about 29% cpu, Obviously, it
> has been the bottleneck in 100% scan case.
> See the [^hbase20-ssd-100-scan-traces.svg]
> BTW, in our XiaoMi branch, we introduce a
> HRegion#updateReadRequestsByCapacityUnitPerSecond to sum up the size of cells
> (for metric monitor), so it seems the performance loss was amplified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21588) Procedure v2 wal splitting implementation
[ https://issues.apache.org/jira/browse/HBASE-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733916#comment-16733916 ] Jingyun Tian commented on HBASE-21588: -- [ERROR] org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Time elapsed: 24.063 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 780 seconds at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.createTable(TestServerCrashProcedure.java:228) at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.testRecoveryAndDoubleExecution(TestServerCrashProcedure.java:138) at org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure.testCrashTargetRs(TestServerCrashProcedure.java:112) [ERROR] org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Time elapsed: 24.357 s <<< ERROR! According this log, this TestServerCrashProcedure is not stuck but exceeds time limit since it's too large. it costs 10min 52sec on my own desktop. Let me split this UT. > Procedure v2 wal splitting implementation > - > > Key: HBASE-21588 > URL: https://issues.apache.org/jira/browse/HBASE-21588 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21588.master.003.patch, > HBASE-21588.master.004.patch, HBASE-21588.master.005.patch, > HBASE-21588.master.006.patch, HBASE-21588.master.007.patch, > HBASE-21588.master.008.patch, HBASE-21588.master.009.patch, > HBASE-21588.master.010.patch, HBASE-21588.master.011.patch, > HBASE-21588.master.012.patch, HBASE-21588.master.013.patch, > HBASE-21588.master.014.patch, HBASE-21588.master.015.patch, > HBASE-21588.master.016.patch, HBASE-21588.master.017.patch, > HBASE-21588.master.018.patch, HBASE-21588.master.018.patch, > HBASE-21588.master.019.patch > > > create a sub task to submit the implementation of procedure v2 wal splitting -- This message was sent by Atlassian JIRA (v7.6.3#76005)
