in some case class StoreScanner can noly do seekScanner in serial
version : hbase 0.98.10 class: storeScanner method: seekScanners(line 314 to 334) if we add columns to scan, then parameter "isLazy" will be false so we can only do seek in serial, not parallel
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389.patch As [~claraxiong] said, just implement multi file writer and compactor in this issue. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang > Attachments: HBASE-15389-uc.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Assignee: Duo Zhang Fix Version/s: 0.98.19 1.3.0 2.0.0 Status: Patch Available (was: Open) > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182741#comment-15182741 ] Clara Xiong commented on HBASE-15389: - Just a minor thing: DateTieredComparator should be DateTieredCompactor as I changed in the patch I uploaded. My patch does expect this name. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182745#comment-15182745 ] Duo Zhang commented on HBASE-15389: --- Oh, typo, let me fix it. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389-v1.patch > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15406:
--
Assignee: (was: Heng Chen)
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15338) Add a option to disable the data block cache for testing the performance of underlying file system
[ https://issues.apache.org/jira/browse/HBASE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182749#comment-15182749 ] Liu Shaohui commented on HBASE-15338: - Committed to master~ > Add a option to disable the data block cache for testing the performance of > underlying file system > -- > > Key: HBASE-15338 > URL: https://issues.apache.org/jira/browse/HBASE-15338 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15338-trunk-v1.diff, HBASE-15338-trunk-v2.diff, > HBASE-15338-trunk-v3.diff, HBASE-15338-trunk-v4.diff, > HBASE-15338-trunk-v5.diff, HBASE-15338-trunk-v6.diff, > HBASE-15338-trunk-v7.diff > > > When testing and comparing the performance of different file systems(HDFS, > Azure blob storage, AWS S3 and so on) for HBase, it's better to avoid the > affect of the HBase BlockCache and get the actually random read latency when > data block is read from underlying file system. (Usually, the index block and > meta block should be cached in memory in the testing). > So we add a option in CacheConfig to disable the data block cache. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (HBASE-15338) Add a option to disable the data block cache for testing the performance of underlying file system
[ https://issues.apache.org/jira/browse/HBASE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui closed HBASE-15338. --- > Add a option to disable the data block cache for testing the performance of > underlying file system > -- > > Key: HBASE-15338 > URL: https://issues.apache.org/jira/browse/HBASE-15338 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15338-trunk-v1.diff, HBASE-15338-trunk-v2.diff, > HBASE-15338-trunk-v3.diff, HBASE-15338-trunk-v4.diff, > HBASE-15338-trunk-v5.diff, HBASE-15338-trunk-v6.diff, > HBASE-15338-trunk-v7.diff > > > When testing and comparing the performance of different file systems(HDFS, > Azure blob storage, AWS S3 and so on) for HBase, it's better to avoid the > affect of the HBase BlockCache and get the actually random read latency when > data block is read from underlying file system. (Usually, the index block and > meta block should be cached in memory in the testing). > So we add a option in CacheConfig to disable the data block cache. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15408) MiniCluster's master crash on initialization and unittest timeout
[
https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182793#comment-15182793
]
Hadoop QA commented on HBASE-15408:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/latest/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 32s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 35s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 190m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
| JDK v1.7.0_79 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hba
[jira] [Created] (HBASE-15409) TestHFileBackedByBucketCache failed randomly on jdk8
Liu Shaohui created HBASE-15409:
---
Summary: TestHFileBackedByBucketCache failed randomly on jdk8
Key: HBASE-15409
URL: https://issues.apache.org/jira/browse/HBASE-15409
Project: HBase
Issue Type: Test
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
When running the small tests, we found TestHFileBackedByBucketCache failed
randomly
{code}
mvn clean package install -DrunSmallTests -Dtest=TestHFileBackedByBucketCache
Running org.apache.hadoop.hbase.io.hfile.TestHFileBackedByBucketCache
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.262 sec <<<
FAILURE! - in org.apache.hadoop.hbase.io.hfile.TestHFileBackedByBucketCache
testBucketCacheCachesAndPersists(org.apache.hadoop.hbase.io.hfile.TestHFileBackedByBucketCache)
Time elapsed: 0.69 sec <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<4>
at
org.apache.hadoop.hbase.io.hfile.TestHFileBackedByBucketCache.testBucketCacheCachesAndPersists(TestHFileBackedByBucketCache.java:161)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182817#comment-15182817
]
Heng Chen commented on HBASE-15376:
---
push to master, branch-1, branch-1.3
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15376:
--
Resolution: Fixed
Assignee: Heng Chen
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
1.3.0
2.0.0
Status: Resolved (was: Patch Available)
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182820#comment-15182820
]
Heng Chen commented on HBASE-15377:
---
duplicate with HBASE-15376
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen resolved HBASE-15377.
---
Resolution: Duplicate
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[
https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182832#comment-15182832
]
Hadoop QA commented on HBASE-15393:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 53s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 101m 35s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 14s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 241m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests |
hadoop.hbase.regionserver.TestFailedAppendAndSync |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791716/HBASE-15393.branch-1.patch
|
| JIRA Issue | HBASE-15393 |
| Optional Tests | asflicense javac javadoc unit find
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182850#comment-15182850
]
Hadoop QA commented on HBASE-15236:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 43s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 3s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 274m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Timed out junit tests |
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface |
| | org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
| | org.apache.hadoop.hbase.wal.TestWALFiltering |
| | org.apache.hadoop.hbase.wal.TestDefaultWALProviderWithHLogKey |
| | org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver |
| | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
| | org.apache.hadoop.hbase.TestZooKeeper |
|
[jira] [Commented] (HBASE-15408) MiniCluster's master crash on initialization and unittest timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182875#comment-15182875 ] Phil Yang commented on HBASE-15408: --- Still timeout and it seems be caused by ScannerCallableWithReplicas. Digging into it. > MiniCluster's master crash on initialization and unittest timeout > - > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408.v1.txt, fail.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crash on initialization > and minicluster will log "No master found; retry" forever. The crash is > caused by OutOfOrderScannerNextException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182878#comment-15182878
]
Hudson commented on HBASE-15376:
SUCCESS: Integrated in HBase-1.3-IT #540 (See
[https://builds.apache.org/job/HBase-1.3-IT/540/])
HBASE-15376 ScanNext metric is size-based while every other (chenheng: rev
bbb10c4c18ec671f161311c5384e73a4a3774d58)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSourceImpl.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15338) Add a option to disable the data block cache for testing the performance of underlying file system
[ https://issues.apache.org/jira/browse/HBASE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182883#comment-15182883 ] Hudson commented on HBASE-15338: FAILURE: Integrated in HBase-Trunk_matrix #760 (See [https://builds.apache.org/job/HBase-Trunk_matrix/760/]) HBASE-15338: Add a option to disable the data block cache for testing (liushaohui: rev 4483b3cbed0c9cc17116af2407d946e605d6d627) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java > Add a option to disable the data block cache for testing the performance of > underlying file system > -- > > Key: HBASE-15338 > URL: https://issues.apache.org/jira/browse/HBASE-15338 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15338-trunk-v1.diff, HBASE-15338-trunk-v2.diff, > HBASE-15338-trunk-v3.diff, HBASE-15338-trunk-v4.diff, > HBASE-15338-trunk-v5.diff, HBASE-15338-trunk-v6.diff, > HBASE-15338-trunk-v7.diff > > > When testing and comparing the performance of different file systems(HDFS, > Azure blob storage, AWS S3 and so on) for HBase, it's better to avoid the > affect of the HBase BlockCache and get the actually random read latency when > data block is read from underlying file system. (Usually, the index block and > meta block should be cached in memory in the testing). > So we add a option in CacheConfig to disable the data block cache. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14498) Master stuck in infinite loop when all Zookeeper servers are unreachable
[
https://issues.apache.org/jira/browse/HBASE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182890#comment-15182890
]
Pankaj Kumar commented on HBASE-14498:
--
Not related test case failure.
> Master stuck in infinite loop when all Zookeeper servers are unreachable
>
>
> Key: HBASE-14498
> URL: https://issues.apache.org/jira/browse/HBASE-14498
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14498-V2.patch, HBASE-14498-V3.patch,
> HBASE-14498-V4.patch, HBASE-14498-V5.patch, HBASE-14498-V6.patch,
> HBASE-14498.patch
>
>
> We met a weird scenario in our production environment.
> In a HA cluster,
> > Active Master (HM1) is not able to connect to any Zookeeper server (due to
> > N/w breakdown on master machine network with Zookeeper servers).
> {code}
> 2015-09-26 15:24:47,508 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 33463ms for sessionid 0x104576b8dda0002, closing socket connection and
> attempting reconnect
> 2015-09-26 15:24:47,877 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:48,236 INFO [main-SendThread(ZK-Host1:2181)]
> client.FourLetterWordMain: connecting to ZK-Host1 2181
> 2015-09-26 15:24:49,879 WARN
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:49,879 INFO
> [HM1-Host:16000.activeMasterManager-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-IP1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:24:50,238 WARN [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host1
> 2015-09-26 15:24:50,238 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> ZK-Host1/ZK-Host1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2015-09-26 15:25:17,470 INFO [main-SendThread(ZK-Host1:2181)]
> zookeeper.ClientCnxn: Client session timed out, have not heard from server in
> 30023ms for sessionid 0x2045762cc710006, closing socket connection and
> attempting reconnect
> 2015-09-26 15:25:17,571 WARN [master/HM1-Host/HM1-IP:16000]
> zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper,
> quorum=ZK-Host:2181,ZK-Host1:2181,ZK-Host2:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2015-09-26 15:25:17,872 INFO [main-SendThread(ZK-Host:2181)]
> client.FourLetterWordMain: connecting to ZK-Host 2181
> 2015-09-26 15:25:19,874 WARN [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Can not get the principle name from server ZK-Host
> 2015-09-26 15:25:19,874 INFO [main-SendThread(ZK-Host:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server ZK-Host/ZK-IP:2181.
> Will not attempt to authenticate using SASL (unknown error)
> {code}
> > Since HM1 was not able to connect to any ZK, so session timeout didnt
> > happen at Zookeeper server side and HM1 didnt abort.
> > On Zookeeper session timeout standby master (HM2) registered himself as an
> > active master.
> > HM2 is keep on waiting for region server to report him as part of active
> > master intialization.
> {noformat}
> 2015-09-26 15:24:44,928 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 0 ms,
> expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval
> of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> ---
> ---
> 2015-09-26 15:32:50,841 | INFO | HM2-Host:21300.activeMasterManager | Waiting
> for region servers count to settle; currently checked in 0, slept for 483913
> ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms,
> interval of 1500 ms. |
> org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011)
> {noformat}
> > At other end, region servers are reporting to HM1 on 3 sec interval. Here
> > region server retrieve master location from zookeeper only when they
> > couldn't connect to Master (ServiceException).
> Region Server will not report HM2 as per current design until unless HM1
> abort,so HM2 will exit(InitializationMonitor) and again wait for region
> servers in loop.
--
This message wa
[jira] [Updated] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15398: -- Attachment: HBASE-15398.v1.txt TestPartialResultsFromClientSide can pass locally. Have a test to see if other tests pass. > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15398: -- Status: Patch Available (was: Open) > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.1.3, 1.2.0 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[
https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182912#comment-15182912
]
Hadoop QA commented on HBASE-15393:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} 0.98 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} 0.98 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} 0.98 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s
{color} | {color:red} hbase-client in 0.98 has 19 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 34s
{color} | {color:red} hbase-server in 0.98 has 82 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hbase-client in 0.98 failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 34s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} 0.98 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 9m
28s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0
2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 117m 41s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 24s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 120m 32s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s {color} | {color:green} Patch does not generate ASF Li
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182916#comment-15182916 ] Duo Zhang commented on HBASE-15389: --- https://github.com/Apache9/hbase/commit/e2fd976bb9cf6439847a2833cd291e3331cb0c8a This is the patch for 0.98. [~claraxiong] you can use it to test your patch for 0.98. Will upload 0.98 patch here when we finish reviewing the patch for master. Thanks. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182963#comment-15182963
]
Hadoop QA commented on HBASE-15389:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 25s
{color} | {color:red} Patch generated 3 new checkstyle issues in hbase-server
(total was 8, now 9). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 28s
{color} | {color:red} hbase-server introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 125m 26s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 20s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 283m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Dead store to readersToClose in
org.apache.hadoop.hbase.regionserver.compactions.DateTieredComparator.compact(CompactionRequest,
List, ThroughputController, User) At
DateTieredComparator.java:org.apache.hadoop.hbase.regionserver.compactions.DateTieredComparator.compact(CompactionRequest,
List, ThroughputController, User) At DateTieredComparator.java:[line 84] |
| JDK v1.8.0 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apac
[jira] [Updated] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15393: --- Fix Version/s: (was: 1.0.4) > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389-v2.patch Addressing the findbugs warning. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389-v2.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15393: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 1.1.4) Status: Resolved (was: Patch Available) Thanks for the patch, Ashish. > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182988#comment-15182988
]
Ted Yu commented on HBASE-15398:
{code}
174 if(!filter.isFamilyEssential(family)){
175 return true;
{code}
Is the above condition inverted ?
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182990#comment-15182990
]
Phil Yang commented on HBASE-15398:
---
I check the comment in:
{code}
/**
* Check that given column family is essential for filter to check row. Most
filters always return
* true here. But some could have more sophisticated logic which could
significantly reduce
* scanning process by not even touching columns until we are 100% sure that
it's data is needed
* in result.
*
* Concrete implementers can signal a failure condition in their code by
throwing an
* {@link IOException}.
*
* @throws IOException in case an I/O or an filter specific failure needs to
be signaled.
*/
abstract public boolean isFamilyEssential(byte[] name) throws IOException;
{code}
and in HRegion
{code}
if (this.filter == null || !scan.doLoadColumnFamiliesOnDemand()
|| this.filter.isFamilyEssential(entry.getKey())) {
scanners.add(scanner);
} else {
joinedScanners.add(scanner);
}
{code}
So if we want the family be filtered, isFamilyEssential return true, right? And
if there is a cf that return false, we will have two heaps in RegionScanner so
we should make checkEssentialFilter return true?
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15408) MiniCluster's master crash on initialization and unittest timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15408: -- Attachment: fail2.txt Find another case resulting in the master crash during test. Seems that we send a request of merging regions but the region is opening. Curiously why we can not handle this exception correctly? > MiniCluster's master crash on initialization and unittest timeout > - > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408.v1.txt, fail.txt, fail2.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crash on initialization > and minicluster will log "No master found; retry" forever. The crash is > caused by OutOfOrderScannerNextException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15408) MiniCluster's master crash on initialization and unittest timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15408: -- Attachment: HBASE-15408-v2.patch > MiniCluster's master crash on initialization and unittest timeout > - > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408-v2.patch, HBASE-15408.v1.txt, fail.txt, > fail2.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crash on initialization > and minicluster will log "No master found; retry" forever. The crash is > caused by OutOfOrderScannerNextException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15408: -- Summary: MiniCluster's master crashes and unit tests timeout (was: MiniCluster's master crash on initialization and unittest timeout) > MiniCluster's master crashes and unit tests timeout > --- > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408-v2.patch, HBASE-15408.v1.txt, fail.txt, > fail2.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crash on initialization > and minicluster will log "No master found; retry" forever. The crash is > caused by OutOfOrderScannerNextException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183006#comment-15183006 ] Phil Yang commented on HBASE-15408: --- Add a throwable catching on DispatchMergingRegionHandler.process() > MiniCluster's master crashes and unit tests timeout > --- > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408-v2.patch, HBASE-15408.v1.txt, fail.txt, > fail2.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crash on initialization > and minicluster will log "No master found; retry" forever. The crash is > caused by OutOfOrderScannerNextException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15408: -- Description: These days there are many tests timeout on builds.apache.org. I have no log on timeout tests but I find a possible reason: master crashes and MiniCluster will log "No master found; retry" forever until timeout. (was: These days there are many tests timeout on builds.apache.org. I have no log on timeout tests but I find a possible reason: master crash on initialization and minicluster will log "No master found; retry" forever. The crash is caused by OutOfOrderScannerNextException.) > MiniCluster's master crashes and unit tests timeout > --- > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408-v2.patch, HBASE-15408.v1.txt, fail.txt, > fail2.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crashes and MiniCluster > will log "No master found; retry" forever until timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10605) Manage the call timeout in the server
[ https://issues.apache.org/jira/browse/HBASE-10605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183013#comment-15183013 ] Nicolas Liochon commented on HBASE-10605: - > , I think client rpc should include the timeout parameter Yes, we would need to forward the timeout (not the the submit time, because we don't want to rely on having the server and client clocks in sync: the server can use its own clock). Then there is already a check in the server, the request is cancelled if the client is disconnected (i.e. the tcp connection is closed). > Manage the call timeout in the server > - > > Key: HBASE-10605 > URL: https://issues.apache.org/jira/browse/HBASE-10605 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.99.0 >Reporter: Nicolas Liochon > > Since HBASE-10566, we have an explicit call timeout available in the client. > We could forward it to the server, and use this information for: > - if the call is still in the queue, just cancel it > - if the call is under execution, makes this information available in > RpcCallContext (actually change the RpcCallContext#disconnectSince to > something more generic), so it can be used by the query under execution to > stop its execution > - in the future, interrupt it to manage the case 'stuck on a dead datanode' > or something similar > - if the operation has finished, don't send the reply to the client, as by > definition the client is not interested anymore. > From this, it will be easy to manage the cancellation: > disconnect/timeout/cancellation are similar from a service execution PoV -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183041#comment-15183041
]
Hadoop QA commented on HBASE-15389:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 7m 21s
{color} | {color:red} Patch generated 3 new checkstyle issues in hbase-server
(total was 8, now 9). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
49m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 5s
{color} | {color:red} hbase-server introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 120m 18s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 42s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 329m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Dead store to readersToClose in
org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(CompactionRequest,
List, ThroughputController, User) At
DateTieredCompactor.java:org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(CompactionRequest,
List, ThroughputController, User) At DateTieredCompactor.java:[line 84] |
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | hadoop.hbase.io.hfile.TestCacheConfig |
| JDK v1.8.0 Timed out junit tests |
org.apache.hadoop.h
[jira] [Updated] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[ https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15408: -- Attachment: HBASE-15408-v2.1.patch Use a subclass of HMaster overriding abort() to print log for HMaster crashing. Will fail some tests. > MiniCluster's master crashes and unit tests timeout > --- > > Key: HBASE-15408 > URL: https://issues.apache.org/jira/browse/HBASE-15408 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-15408-v2.1.patch, HBASE-15408-v2.patch, > HBASE-15408.v1.txt, fail.txt, fail2.txt > > > These days there are many tests timeout on builds.apache.org. I have no log > on timeout tests but I find a possible reason: master crashes and MiniCluster > will log "No master found; retry" forever until timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183072#comment-15183072
]
Hudson commented on HBASE-15376:
FAILURE: Integrated in HBase-1.4 #4 (See
[https://builds.apache.org/job/HBase-1.4/4/])
HBASE-15376 ScanNext metric is size-based while every other (chenheng: rev
bbb10c4c18ec671f161311c5384e73a4a3774d58)
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSourceImpl.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183088#comment-15183088 ] Hudson commented on HBASE-15393: SUCCESS: Integrated in HBase-1.3-IT #541 (See [https://builds.apache.org/job/HBase-1.3-IT/541/]) HBASE-15393 Enable table replication command will fail when parent znode (tedyu: rev c1524485c58ad4fea8849e1e3dca4bf4e78e1034) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithTwoDifferentZKClusters.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183087#comment-15183087
]
Hudson commented on HBASE-15376:
FAILURE: Integrated in HBase-1.3 #589 (See
[https://builds.apache.org/job/HBase-1.3/589/])
HBASE-15376 ScanNext metric is size-based while every other (chenheng: rev
69fcbd825aa6c487d2bdfebf774636ae4685d74d)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSourceImpl.java
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6014) Support for block-granularity bitmap indexes
[ https://issues.apache.org/jira/browse/HBASE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183104#comment-15183104 ] Daniel Lemire commented on HBASE-6014: -- A good choice for this might be Roaring bitmaps (http://roaringbitmap.org/). They are used by Apache Spark, Druid, Apache Kylin, Apache Lucene and so forth. No patent, Apache license. > Support for block-granularity bitmap indexes > > > Key: HBASE-6014 > URL: https://issues.apache.org/jira/browse/HBASE-6014 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Todd Lipcon > Attachments: 6014-bitmap-hacking.txt, bitmap-hacking.txt > > > This came up in a discussion with Kannan today, so I promised to write > something brief on JIRA -- this was suggested as a potential summer intern > project. The idea is as follows: > We have several customers who periodically run full table scan MR jobs > against large HBase tables while applying fairly restrictive predicates. The > predicates are often reasonably simple boolean expressions across known > columns, and those columns often are enum-typed or otherwise have a fairly > restricted range of values. For example, a real time process may mark rows as > dirty, and a background MR job may scan for dirty rows in order to perform > further processing like rebuilding inverted indexes. > One way to speed up this type of query is to add bitmap indexes. In the > context of HBase, I would envision this as a new type of metadata block > included in the HFile which has a series of tuples: (qualifier, value range, > compressed bitmap). A 1 bit in the bitmap indicates that the corresponding > HFile block has at least one cell for which a column with the given qualifier > falls within the given range. Queries which have an equality or comparison > predicate against an indexed qualifier can then use the bitmap index to seek > directly to those blocks which may contain relevant data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[
https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183111#comment-15183111
]
Hadoop QA commented on HBASE-15408:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 31s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 26s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791760/HBASE-15408-v2.patch |
| JIRA Issue | HBASE-15408 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3
[jira] [Commented] (HBASE-15243) Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183130#comment-15183130 ] Ted Yu commented on HBASE-15243: Thanks for the review, Lars. If there is no further review comment, I will integrate later today. > Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList > return SEEK_NEXT_USING_HINT > -- > > Key: HBASE-15243 > URL: https://issues.apache.org/jira/browse/HBASE-15243 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15243-v1.txt, HBASE-15243-v2.txt, > HBASE-15243-v3.txt, HBASE-15243-v4.txt, HBASE-15243-v5.txt, > HBASE-15243-v6.txt, HBASE-15243-v7.txt > > > As Preston Koprivica pointed out at the tail of HBASE-4394, when all filters > in a MUST_PASS_ONE FilterList return a SEEK_USING_NEXT_HINT code, we should > return SEEK_NEXT_USING_HINT from the FilterList to utilize the lowest seek > value. > This would save unnecessary accesses to blocks which are not in scan result. > A unit test has been added which shows the reduction in the number of blocks > accessed when this optimization takes effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183151#comment-15183151
]
Ted Yu commented on HBASE-15398:
The return value of checkEssentialFilter() is assigned to:
{code}
+protected boolean hasEssentialFilter = false;
{code}
whose default value is false.
Inside the method, if filter.isFamilyEssential() returns true, for loop
continues. Meaning, for a multi column family table, checkEssentialFilter()
would likely return false since not all column families are essential.
Is the above what you wanted ?
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183158#comment-15183158 ] Hudson commented on HBASE-15393: SUCCESS: Integrated in HBase-1.2-IT #455 (See [https://builds.apache.org/job/HBase-1.2-IT/455/]) HBASE-15393 Enable table replication command will fail when parent znode (tedyu: rev 66e2c8ddac4ad5df8e10ea785b8b8a129b08cfae) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithTwoDifferentZKClusters.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183187#comment-15183187 ] Phil Yang commented on HBASE-15398: --- I expected that if there is a family need not be filtered(not essential), checkEssentialFilter() return true. So if all cf should be filtered or there is no filter, checkEssentialFilter() should return false. And if hasEssentialFilter is true, we should sort in createCompleteResult and not return partial results (or throw an exception). I realized that call them checkEssentialFilter and hasEssentialFilter is confusing... We need a better name here... hasUnenssentialFamily? Could you give me any suggest here? > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183197#comment-15183197 ] Hudson commented on HBASE-15393: FAILURE: Integrated in HBase-Trunk_matrix #761 (See [https://builds.apache.org/job/HBase-Trunk_matrix/761/]) HBASE-15393 Addendum adds TestReplicationAdminWithTwoDifferentZKClusters (tedyu: rev 05161fcbfdd78f5684b9cb52c49a02be5ad14499) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithTwoDifferentZKClusters.java > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183198#comment-15183198
]
Hudson commented on HBASE-15376:
FAILURE: Integrated in HBase-Trunk_matrix #761 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/761/])
HBASE-15376 ScanNext metric is size-based while every other (chenheng: rev
f30afa05d9274af2301ff7ffdbd8d6e42599ef57)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSourceImpl.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionSource.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegion.java
*
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
*
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183228#comment-15183228 ] Sean Busbey commented on HBASE-14703: - I'd like to avoid the change on branch-1.2. It's too easy to violate the assumption that mixed patch versions would be brief. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183264#comment-15183264
]
Ted Yu commented on HBASE-15406:
bq. when one presses ctrl-c in terminal, that should be interruptible signal
I wouldn't depend on the signal handler for cleaning up the switch. As
mentioned earlier, network partition may render the cleanup ineffective.
If we introduce ephemeral znode, do we still keep the current switch znode ?
If so, changing the switch from hbck would involve adjusting value of current
znode and adding ephemeral znode at the same time. The values stored in current
znode would need to expand beyond true / false because we want to distinguish
the scenario where hbck (or some other tool) turns off split / merge for the
duration of the operation.
What do you think, [~mantonov] ?
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[
https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183280#comment-15183280
]
Hadoop QA commented on HBASE-15408:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 57s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 49s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hbase.master.handler.TestCreateTableHandler |
| | hadoop.hbase.master.TestMasterFailover |
| | hadoop.hbase.io.hfile.TestCacheConfig |
| | hadoop.hbase.master.TestDistributedLogSplitting |
| | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| JDK v1.7.0_79 Timed out junit tests |
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface |
| | org.apache.hadoop.hbase.coprocessor.TestHTableWrapper |
| | org.apache.hadoop.hbase.mapreduce.TestTableInputFormat |
| | org.apache.hadoop.hbase.snapshot.TestSnaps
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183285#comment-15183285
]
Hadoop QA commented on HBASE-15398:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/latest/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 11m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 45s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m 44s
{color} | {color:red} Patch generated 21 new checkstyle issues in hbase-client
(total was 221, now 242). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 7m 51s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 214, now 215). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
61m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 36s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 138m 19s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 17s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 7s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} Patch does not g
[jira] [Updated] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-15236:
-
Status: Open (was: Patch Available)
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-15236:
-
Status: Patch Available (was: Open)
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15390) ClientExceptionUtils doesn't handle CallQueueTooBigException properly
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183313#comment-15183313 ] Sean Busbey commented on HBASE-15390: - What's the path forward here? > ClientExceptionUtils doesn't handle CallQueueTooBigException properly > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0, 1.2.1 > > > In #isMetaClearingException() we're checking for CallQueueTooBigException, > but under debugger in tests I saw that what we're really getting in here is > RemoteWithExtrasException, which doesn't allow to easily unwrap CQTBE from > it, since it's stored in the classname field, and findException() or > unwrapRemoteException() fail to unwrap it correctly. > I think we'd have the same behavior with other exceptions wrapper this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15408) MiniCluster's master crashes and unit tests timeout
[
https://issues.apache.org/jira/browse/HBASE-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183325#comment-15183325
]
Hadoop QA commented on HBASE-15408:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 8s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 1s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 224m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.io.hfile.TestCacheConfig |
| JDK v1.8.0 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
| | org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| | org.apache.hadoop.hbase.wal.TestWALFiltering |
| | org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver |
| | org.apache.hadoop.hbase.wal.TestDef
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183334#comment-15183334
]
Ted Yu commented on HBASE-15398:
bq. And if hasEssentialFilter is true, we should sort in createCompleteResult
When filter is present, in normal cases, there would be no essential column
family, leading to checkEssentialFilter() returning true. Why do we need extra
sort in this scenario ?
{code}
-if (allowPartials || isBatchSet) {
+if (!hasEssentialFilter && (allowPartials || isBatchSet)) {
{code}
If user specifies allowPartials, that should be respected. Why is the condition
guarded by hasEssentialFilter ?
I don't see new exception added which would alert the user that partial result
wouldn't be delivered.
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15410) Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList return SEEK_NEXT_USING_HINT
Ted Yu created HBASE-15410: -- Summary: Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList return SEEK_NEXT_USING_HINT Key: HBASE-15410 URL: https://issues.apache.org/jira/browse/HBASE-15410 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu As Preston mentioned in the comment in HBASE-15243: https://issues.apache.org/jira/browse/HBASE-15243?focusedCommentId=15143557&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15143557 an optimization for filters returning a SEEK_NEXT_USING_HINT would be to seek to the highest hint (Any previous/lower row won't be accepted by the filter returning that seek). This JIRA is to explore this potential optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15410) Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList return SEEK_NEXT_USING_HINT
[ https://issues.apache.org/jira/browse/HBASE-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15410: --- Attachment: 15410-wip.patch > Utilize the max seek value when all Filters in MUST_PASS_ALL FilterList > return SEEK_NEXT_USING_HINT > --- > > Key: HBASE-15410 > URL: https://issues.apache.org/jira/browse/HBASE-15410 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15410-wip.patch > > > As Preston mentioned in the comment in HBASE-15243: > https://issues.apache.org/jira/browse/HBASE-15243?focusedCommentId=15143557&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15143557 > an optimization for filters returning a SEEK_NEXT_USING_HINT would be to seek > to the highest hint (Any previous/lower row won't be accepted by the filter > returning that seek). > This JIRA is to explore this potential optimization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[
https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183355#comment-15183355
]
Hadoop QA commented on HBASE-15389:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 9s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 8, now 8). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 21s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 52s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 283m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.io.hfile.TestCacheConfig |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.io.hfile.TestCacheConfig |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791756/HBASE-15389-v2.patch |
| JIRA Issue | HBASE-15389 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux proserpina.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jen
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183356#comment-15183356
]
Dave Latham commented on HBASE-15236:
-
This sounds like one bad outcome that can happen if there are multiple HFiles
with the same seq id, and tries to address it. However, I think there are
others - for example compacting some of those files could end up also changing
the view of the data. I wonder if we'd be better served by not allowing such a
condition.
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183381#comment-15183381 ] Hudson commented on HBASE-15393: SUCCESS: Integrated in HBase-1.4 #5 (See [https://builds.apache.org/job/HBase-1.4/5/]) HBASE-15393 Enable table replication command will fail when parent znode (tedyu: rev c1524485c58ad4fea8849e1e3dca4bf4e78e1034) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithTwoDifferentZKClusters.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183398#comment-15183398 ] Andrew Purtell commented on HBASE-15130: I'd like to revert this from 0.98.18. We could try again for 0.98.19 with a backport that does not break Phoenix. Any objection to a revert? If not I will commit it later today. > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.18 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15405) Fix PE logging and wrong defaults in help message
[
https://issues.apache.org/jira/browse/HBASE-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183418#comment-15183418
]
Appy commented on HBASE-15405:
--
Ready for review.
> Fix PE logging and wrong defaults in help message
> -
>
> Key: HBASE-15405
> URL: https://issues.apache.org/jira/browse/HBASE-15405
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-15405-master.patch, HBASE-15405-master.patch
>
>
> Corrects wrong default values for few options in the help message.
> Final stats from multiple clients are intermingled making it hard to
> understand. Also the logged stats aren't very machine readable. It can be
> helpful in a daily perf testing rig which scraps logs for results.
> Example of logs before the change.
> {noformat}
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.98, min=359.00, max=324050.00, stdDev=851.82, 95th=1368.00,
> 99th=1625.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.92, min=356.00, max=323394.00, stdDev=817.55, 95th=1370.00,
> 99th=1618.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: 0/1048570/1048576,
> latency mean=953.98, min=367.00, max=322745.00, stdDev=840.43, 95th=1369.00,
> 99th=1622.00
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest latency log
> (microseconds), on 1048576 measures
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 375.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 363.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.6624126434326
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.4124526977539
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 781.3929776087633
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 742.8027916717297
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 894.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1070.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1071.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th =
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 95th =
> 1369.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th =
> 1623.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99th =
> 1624.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Min =
> 372.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th =
> 3013.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest Avg =
> 953.2451229095459
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.9th =
> 3043.998000214
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest StdDev =
> 725.4744472152282
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th =
> 25282.38016755
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 50th =
> 895.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.99th =
> 25812.76334
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 75th =
> 1071.0
> 16/03/05 22:43:06 INFO hbase.PerformanceEvaluation: IncrementTest 99.999th =
> 89772
[jira] [Updated] (HBASE-15300) Upgrade to zookeeper 3.4.8
[ https://issues.apache.org/jira/browse/HBASE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15300: --- Attachment: HBASE-15300.v2.patch Patch v2 includes a trivial change in NamespaceAuditor.java so that QA runs all tests. > Upgrade to zookeeper 3.4.8 > -- > > Key: HBASE-15300 > URL: https://issues.apache.org/jira/browse/HBASE-15300 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Labels: zookeeper > Attachments: HBASE-15300.v1.patch, HBASE-15300.v2.patch, > HBASE-1533.v1.patch > > > zookeeper 3.4.8 has been released. > Among the fixes the following are desirable: > ZOOKEEPER-706 large numbers of watches can cause session re-establishment to > fail > ZOOKEEPER-1797 PurgeTxnLog may delete data logs during roll > This issue upgrades zookeeper dependency to 3.4.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183442#comment-15183442
]
Appy commented on HBASE-15236:
--
I am sure such a restriction will make things simpler and am in favor of it.
But what if a user creates hfiles throughout the day and bulkloads them once
when cluster is under low load?
I think [~mbertozzi] might know other cases too.
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183518#comment-15183518 ] Hudson commented on HBASE-15393: SUCCESS: Integrated in HBase-1.2 #570 (See [https://builds.apache.org/job/HBase-1.2/570/]) HBASE-15393 Enable table replication command will fail when parent znode (tedyu: rev 66e2c8ddac4ad5df8e10ea785b8b8a129b08cfae) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithTwoDifferentZKClusters.java > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15393) Enable table replication command will fail when parent znode is not default in peer cluster
[ https://issues.apache.org/jira/browse/HBASE-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183533#comment-15183533 ] Hudson commented on HBASE-15393: FAILURE: Integrated in HBase-1.3 #590 (See [https://builds.apache.org/job/HBase-1.3/590/]) HBASE-15393 Enable table replication command will fail when parent znode (tedyu: rev 88f62b1db78704e2d67c4e27681d0e92905b74f4) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdminWithTwoDifferentZKClusters.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java > Enable table replication command will fail when parent znode is not default > in peer cluster > --- > > Key: HBASE-15393 > URL: https://issues.apache.org/jira/browse/HBASE-15393 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.0.3, 0.98.17 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.2.1, 0.98.18 > > Attachments: HBASE-15393.0.98.patch, HBASE-15393.branch-1.patch, > HBASE-15393.patch, HBASE-15393.v1.patch, HBASE-15393.v2.patch, > HBASE-15393.v3.patch > > > Enable table replication command will fail when parent znode is not > /hbase(default) in peer cluster and there is only one peer cluster added in > the source cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14030: --- Attachment: HBASE-14030.v38.patch Patch v38 contains a small refactoring (discussed with Vlad offline) where copy service doesn't rely on BackupHandler to get to BackupContext and BackupManager. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v35.patch, > HBASE-14030-v37.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch, > HBASE-14030.v38.patch, hbase-14030_v36.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183551#comment-15183551
]
Dave Latham commented on HBASE-15236:
-
I think bulk load should assign seq ids in a way to prevent duplicates.
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15411) Rewrite backup with Procedure V2
Ted Yu created HBASE-15411: -- Summary: Rewrite backup with Procedure V2 Key: HBASE-15411 URL: https://issues.apache.org/jira/browse/HBASE-15411 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Currently full / incremental backup is driven by BackupHandler (see call() method for flow). This issue is to rewrite the flow using Procedure V2. States (enum) for full / incremental backup would be introduced in Backup.proto which correspond to the steps performed in BackupHandler#call(). executeFromState() would pace the backup based on the current state. serializeStateData() / deserializeStateData() would be used to persist state into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183579#comment-15183579
]
Mikhail Antonov commented on HBASE-15406:
-
[~te...@apache.org] now as I'm thinking about it, I think we may just want to
have ephemeral znodes for the running services (i.e. hbck is in progress,
backup is in progress etc) and a chore which watches those znodes and flips
internal boolean flags on master.
What do others think?
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15322) HBase 1.1.3 crashing
[
https://issues.apache.org/jira/browse/HBASE-15322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183578#comment-15183578
]
Hadoop QA commented on HBASE-15322:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 8s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 10s
{color} | {color:red} hbase-common: patch generated 3 new + 558 unchanged - 0
fixed = 561 total (was 558) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} hbase-common: patch generated 3 new + 558 unchanged - 0
fixed = 561 total (was 558) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} hbase-client: patch generated 3 new + 558 unchanged - 0
fixed = 561 total (was 558) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} hbase-client: patch generated 3 new + 558 unchanged - 0
fixed = 561 total (was 558) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {c
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183585#comment-15183585
]
Mikhail Antonov commented on HBASE-15406:
-
cc [~enis] [~mbertozzi]
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[ https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15261: -- Attachment: HBASE-15261-001.patch Retry +1 on patch. Needs to be volatile. Hard to make a test of concurrent reads when split > Make Throwable t in DaughterOpener volatile > --- > > Key: HBASE-15261 > URL: https://issues.apache.org/jira/browse/HBASE-15261 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-15261-001.patch, HBASE-15261-001.patch > > > In the region split process, daughter regions are opened in different > threads, Throwable t is set in these threads and it is checked in the calling > thread. Need to make it volatile so the checking will not miss any exceptions > from opening daughter regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15390) ClientExceptionUtils doesn't handle CallQueueTooBigException properly
[ https://issues.apache.org/jira/browse/HBASE-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183610#comment-15183610 ] Mikhail Antonov commented on HBASE-15390: - I have a patch for that, I just need to decouple it from other related changes related to MetaCache lookups and post here. That should be good to go in 1.2+ I believe. > ClientExceptionUtils doesn't handle CallQueueTooBigException properly > - > > Key: HBASE-15390 > URL: https://issues.apache.org/jira/browse/HBASE-15390 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.2.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Blocker > Fix For: 1.3.0, 1.2.1 > > > In #isMetaClearingException() we're checking for CallQueueTooBigException, > but under debugger in tests I saw that what we're really getting in here is > RemoteWithExtrasException, which doesn't allow to easily unwrap CQTBE from > it, since it's stored in the classname field, and findException() or > unwrapRemoteException() fail to unwrap it correctly. > I think we'd have the same behavior with other exceptions wrapper this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14411) Fix unit test failures when using multiwal as default WAL provider
[
https://issues.apache.org/jira/browse/HBASE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14411:
---
Attachment: FullTableBackupProcedure.java
Work in progress procedure for full backup
> Fix unit test failures when using multiwal as default WAL provider
> --
>
> Key: HBASE-14411
> URL: https://issues.apache.org/jira/browse/HBASE-14411
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: FullTableBackupProcedure.java,
> HBASE-14411.branch-1.patch, HBASE-14411.patch, HBASE-14411_v2.patch
>
>
> If we set hbase.wal.provider to multiwal in
> hbase-server/src/test/resources/hbase-site.xml which allows us to use
> BoundedRegionGroupingProvider in UT, we will observe below failures in
> current code base:
> {noformat}
> Failed tests:
> TestHLogRecordReader>TestWALRecordReader.testPartialRead:164 expected:<1>
> but was:<2>
> TestHLogRecordReader>TestWALRecordReader.testWALRecordReader:216
> expected:<2> but was:<3>
> TestWALRecordReader.testPartialRead:164 expected:<1> but was:<2>
> TestWALRecordReader.testWALRecordReader:216 expected:<2> but was:<3>
> TestDistributedLogSplitting.testRecoveredEdits:276 edits dir should have
> more than a single file in it. instead has 1
> TestAtomicOperation.testMultiRowMutationMultiThreads:499 expected:<0> but
> was:<1>
> TestHRegionServerBulkLoad.testAtomicBulkLoad:307
> Expected: is
> but: was
> TestLogRolling.testCompactionRecordDoesntBlockRolling:611 Should have WAL;
> one table is not flushed expected:<1> but was:<0>
> TestLogRolling.testLogRollOnDatanodeDeath:359 null
> TestLogRolling.testLogRollOnPipelineRestart:472 Missing datanode should've
> triggered a log roll
> TestReplicationSourceManager.testLogRoll:237 expected:<6> but was:<7>
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594 if
> skip.errors is false all files should remain in place expected:<11> but
> was:<12>
> TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong number of files in the
> archive log expected:<11> but was:<12>
> TestWALSplit.testMovedWALDuringRecovery:810->retryOverHdfsProblem:793
> expected:<11> but was:<12>
> TestWALSplit.testRetryOpenDuringRecovery:838->retryOverHdfsProblem:793
> expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594
> if skip.errors is false all files should remain in place expected:<11> but
> was:<12>
> TestWALSplitCompressed>TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong
> number of files in the archive log expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testMovedWALDuringRecovery:810->TestWALSplit.retryOverHdfsProblem:793
> expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testRetryOpenDuringRecovery:838->TestWALSplit.retryOverHdfsProblem:793
> expected:<11> but was:<12>
> {noformat}
> While patch for HBASE-14306 could resolve failures of TestHLogRecordReader,
> TestReplicationSourceManager and TestReplicationWALReaderManager, this JIRA
> will focus on resolving the others
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14411) Fix unit test failures when using multiwal as default WAL provider
[
https://issues.apache.org/jira/browse/HBASE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14411:
---
Attachment: (was: FullTableBackupProcedure.java)
> Fix unit test failures when using multiwal as default WAL provider
> --
>
> Key: HBASE-14411
> URL: https://issues.apache.org/jira/browse/HBASE-14411
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14411.branch-1.patch, HBASE-14411.patch,
> HBASE-14411_v2.patch
>
>
> If we set hbase.wal.provider to multiwal in
> hbase-server/src/test/resources/hbase-site.xml which allows us to use
> BoundedRegionGroupingProvider in UT, we will observe below failures in
> current code base:
> {noformat}
> Failed tests:
> TestHLogRecordReader>TestWALRecordReader.testPartialRead:164 expected:<1>
> but was:<2>
> TestHLogRecordReader>TestWALRecordReader.testWALRecordReader:216
> expected:<2> but was:<3>
> TestWALRecordReader.testPartialRead:164 expected:<1> but was:<2>
> TestWALRecordReader.testWALRecordReader:216 expected:<2> but was:<3>
> TestDistributedLogSplitting.testRecoveredEdits:276 edits dir should have
> more than a single file in it. instead has 1
> TestAtomicOperation.testMultiRowMutationMultiThreads:499 expected:<0> but
> was:<1>
> TestHRegionServerBulkLoad.testAtomicBulkLoad:307
> Expected: is
> but: was
> TestLogRolling.testCompactionRecordDoesntBlockRolling:611 Should have WAL;
> one table is not flushed expected:<1> but was:<0>
> TestLogRolling.testLogRollOnDatanodeDeath:359 null
> TestLogRolling.testLogRollOnPipelineRestart:472 Missing datanode should've
> triggered a log roll
> TestReplicationSourceManager.testLogRoll:237 expected:<6> but was:<7>
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594 if
> skip.errors is false all files should remain in place expected:<11> but
> was:<12>
> TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong number of files in the
> archive log expected:<11> but was:<12>
> TestWALSplit.testMovedWALDuringRecovery:810->retryOverHdfsProblem:793
> expected:<11> but was:<12>
> TestWALSplit.testRetryOpenDuringRecovery:838->retryOverHdfsProblem:793
> expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594
> if skip.errors is false all files should remain in place expected:<11> but
> was:<12>
> TestWALSplitCompressed>TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong
> number of files in the archive log expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testMovedWALDuringRecovery:810->TestWALSplit.retryOverHdfsProblem:793
> expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testRetryOpenDuringRecovery:838->TestWALSplit.retryOverHdfsProblem:793
> expected:<11> but was:<12>
> {noformat}
> While patch for HBASE-14306 could resolve failures of TestHLogRecordReader,
> TestReplicationSourceManager and TestReplicationWALReaderManager, this JIRA
> will focus on resolving the others
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15411: --- Attachment: FullTableBackupProcedure.java Work in progress full backup procedure. > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183638#comment-15183638
]
Matteo Bertozzi commented on HBASE-15236:
-
the problem is that if you have multiple file per family how do you increment
the sequenceId?
if you have a single family is easy to just increment the sequenceId instead to
using the one we get from flush
but if you have multiple families how do you know how to match the files of
different families?
{code}
public boolean bulkLoadHFiles(Collection> familyPaths,
boolean assignSeqId,
BulkLoadListener bulkLoadListener) throws IOException {
long seqId = -1;
...
if (assignSeqId) {
seqId = ...
}
for (Pair p : familyPaths) {
byte[] familyName = p.getFirst();
String path = p.getSecond();
...
Path commitedStoreFile = store.bulkLoadHFile(finalPath, seqId);
{code}
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1,
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183653#comment-15183653
]
Dave Latham commented on HBASE-15236:
-
Thanks for chiming in, Matteo. I'm definitely coming from a position of
limited knowledge here. I don't quite understand how multiple families affects
the issue. For a single family with multiple files, can we grab a range of
seqIds to assign to them? If there are multiple files, would the same work?
If not, could sequence ids be a per-family concept?
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183657#comment-15183657
]
Hadoop QA commented on HBASE-15236:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 46s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 16s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 19s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.ipc.TestSimpleRpcScheduler |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.io.hfile.TestCacheConfig |
| JDK v1.7.0_79 Timed out junit tests |
org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface |
| | org.apache.hadoop.hbase.mapreduce.TestWALPlayer |
| | org.apache.hadoop.hbase.coprocessor.TestHTableWrapper |
| | org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence |
| | org.apache.hadoop.hbase.mapreduce.TestTableInputFormat |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMultipleTim
[jira] [Commented] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183680#comment-15183680
]
Enis Soztutar commented on HBASE-15376:
---
Thanks [~chenheng]!
Now on to HBASE-15376.
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar reopened HBASE-15376:
---
I think this is not a duplicate. This issue is about the Get metric per region.
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar reopened HBASE-15377:
---
I think this is not a duplicate. This issue is about the Get metric per region.
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15376) ScanNext metric is size-based while every other per-operation metric is time based
[
https://issues.apache.org/jira/browse/HBASE-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-15376.
---
Resolution: Fixed
Ops, reopened wrong issue.
> ScanNext metric is size-based while every other per-operation metric is time
> based
> --
>
> Key: HBASE-15376
> URL: https://issues.apache.org/jira/browse/HBASE-15376
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15376.patch, HBASE-15376_v1.patch,
> HBASE-15376_v3.patch
>
>
> We have per-operation metrics for {{Get}}, {{Mutate}}, {{Delete}},
> {{Increment}}, and {{ScanNext}}.
> The metrics are emitted like:
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> ...
> "ScanNext_num_ops" : 194705,
> "ScanNext_min" : 0,
> "ScanNext_max" : 18441,
> "ScanNext_mean" : 7468.274651395701,
> "ScanNext_median" : 583.0,
> "ScanNext_75th_percentile" : 583.0,
> "ScanNext_95th_percentile" : 13481.0,
> "ScanNext_99th_percentile" : 13481.0,
> {code}
> The problem is that all of Get,Mutate,Delete,Increment,Append,Replay are time
> based tracking how long the operation ran, while ScanNext is tracking
> returned response sizes (returned cell-sizes to be exact). Obviously, this is
> very confusing and you would only know this subtlety if you read the metrics
> collection code.
> Not sure how useful is the ScanNext metric as it is today. We can deprecate
> it, and introduce a time based one to keep track of scan request latencies.
> ps. Shamelessly using the parent jira (since these seem relavant).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15412) Add average region size metric
Enis Soztutar created HBASE-15412: - Summary: Add average region size metric Key: HBASE-15412 URL: https://issues.apache.org/jira/browse/HBASE-15412 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alicia Ying Shu Fix For: 2.0.0, 1.3.0, 1.4.0 We have several metrics related to region store file size, num regions, etc per regionserver, but we do not have a single metric to track the average region size per regionserver. Avg region size is important to look at for deciding on the split policy, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183699#comment-15183699
]
Dave Latham commented on HBASE-15236:
-
the 2nd question should be:
If there are multiple *families*, would the same work?
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: HBASE-14123-v10.patch > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v2.patch, HBASE-14123-v3.patch, HBASE-14123-v4.patch, > HBASE-14123-v5.patch, HBASE-14123-v6.patch, HBASE-14123-v7.patch, > HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15401) Add Zookeeper to third party
[ https://issues.apache.org/jira/browse/HBASE-15401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HBASE-15401: - Assignee: Elliott Clark > Add Zookeeper to third party > > > Key: HBASE-15401 > URL: https://issues.apache.org/jira/browse/HBASE-15401 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Elliott Clark > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
Stephen Yuan Jiang created HBASE-15413: -- Summary: Procedure-V2: print out ProcedureInfo during trace Key: HBASE-15413 URL: https://issues.apache.org/jira/browse/HBASE-15413 Project: HBase Issue Type: Improvement Components: proc-v2 Affects Versions: 1.2.0, 1.3.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Priority: Trivial Before the HBASE-15100 refactored the code to print the ProcedureInfo object, we don't have the needs. Now we need to implement a customized toString() function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-15413: --- Attachment: HBASE-15413.v1-master.patch > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183803#comment-15183803 ] Stephen Yuan Jiang commented on HBASE-15413: [~mbertozzi], FYI. > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15300) Upgrade to zookeeper 3.4.8
[
https://issues.apache.org/jira/browse/HBASE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183806#comment-15183806
]
Hadoop QA commented on HBASE-15300:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 37s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 29s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 31s {color}
| {color:red} root in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 121m 34s
{color} | {color:red} root in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 217m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
| JDK v1.7.0_95 Failed junit tests | hadoop.hbase.io.hfile.TestCacheConfig |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-03-07 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791697/HBASE-15300.v1.patch |
| JIRA Issue | HBASE-15300 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux 70177ef15d79 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 05161fc |
| Default Java | 1.7.0_95 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_74
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 |
| unit |
https://builds.ap
[jira] [Commented] (HBASE-15413) Procedure-V2: print out ProcedureInfo during trace
[ https://issues.apache.org/jira/browse/HBASE-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183824#comment-15183824 ] Matteo Bertozzi commented on HBASE-15413: - can you use the one in HBASE-15113, it contains a couple more info > Procedure-V2: print out ProcedureInfo during trace > -- > > Key: HBASE-15413 > URL: https://issues.apache.org/jira/browse/HBASE-15413 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Affects Versions: 1.2.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Trivial > Attachments: HBASE-15413.v1-master.patch > > > Before the HBASE-15100 refactored the code to print the ProcedureInfo object, > we don't have the needs. Now we need to implement a customized toString() > function to print out information in the ProcedureInfo object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15261) Make Throwable t in DaughterOpener volatile
[
https://issues.apache.org/jira/browse/HBASE-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183834#comment-15183834
]
Hadoop QA commented on HBASE-15261:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 27s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 34s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hbase.io.hfile.TestHFileBackedByBucketCache |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12791813/HBASE-15261-001.patch
|
| JIRA Issue | HBASE-15261 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Se
[jira] [Created] (HBASE-15414) Bound the size of multi request returns and/or allow return of partial result up to client
stack created HBASE-15414: - Summary: Bound the size of multi request returns and/or allow return of partial result up to client Key: HBASE-15414 URL: https://issues.apache.org/jira/browse/HBASE-15414 Project: HBase Issue Type: Improvement Reporter: stack Some knowledgeable hbase users note that while Scanning now allows you return results in 'chunks' for assembly client-side as a whole result (or the application can see the partials as they come out of the cluster), this ability is absent if you do a multi-get; you might get back more than you bargained for and just as chunking when Scanning makes sense because it makes hbase 'regular', we need the same for multiget. Parking an issue here for discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15415) Improve Master WebUI snapshot information
Sean Busbey created HBASE-15415: --- Summary: Improve Master WebUI snapshot information Key: HBASE-15415 URL: https://issues.apache.org/jira/browse/HBASE-15415 Project: HBase Issue Type: Improvement Components: master, snapshots Affects Versions: 0.94.0 Reporter: Sean Busbey Priority: Minor Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.19 On the Master WebUI, we currently show lots of information about the space used by individual snapshots. We should also give a total space used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
