[jira] [Commented] (HBASE-15113) Procedure v2 - Speedup eviction of sys operation results
[ https://issues.apache.org/jira/browse/HBASE-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186691#comment-15186691 ] Hudson commented on HBASE-15113: FAILURE: Integrated in HBase-1.4 #9 (See [https://builds.apache.org/job/HBase-1.4/9/]) HBASE-15113 Procedure v2 - Speedup eviction of sys operation results (matteo.bertozzi: rev 46bf3c32750fd8e03271dd5d8d99918149b928f7) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java > Procedure v2 - Speedup eviction of sys operation results > > > Key: HBASE-15113 > URL: https://issues.apache.org/jira/browse/HBASE-15113 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.2.0, 1.1.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15113-v0.patch, HBASE-15113-v1.patch, > HBASE-15113-v2.patch, HBASE-15113-v3.patch > > > Some system operations like create system tables or namespaces or the server > crash handler. can be evicted sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186692#comment-15186692 ] Hudson commented on HBASE-15422: FAILURE: Integrated in HBase-1.4 #9 (See [https://builds.apache.org/job/HBase-1.4/9/]) HBASE-15422 Procedure v2 - Avoid double yield (matteo.bertozzi: rev c60da6961227f3e78530a877f4a435eb70738b10) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestYieldProcedures.java > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15113) Procedure v2 - Speedup eviction of sys operation results
[ https://issues.apache.org/jira/browse/HBASE-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186667#comment-15186667 ] Hudson commented on HBASE-15113: FAILURE: Integrated in HBase-Trunk_matrix #766 (See [https://builds.apache.org/job/HBase-Trunk_matrix/766/]) HBASE-15113 Procedure v2 - Speedup eviction of sys operation results (matteo.bertozzi: rev 9e967e5c1ddcff46db0b80fb26e1cfe083cd1bfd) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java > Procedure v2 - Speedup eviction of sys operation results > > > Key: HBASE-15113 > URL: https://issues.apache.org/jira/browse/HBASE-15113 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.2.0, 1.1.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15113-v0.patch, HBASE-15113-v1.patch, > HBASE-15113-v2.patch, HBASE-15113-v3.patch > > > Some system operations like create system tables or namespaces or the server > crash handler. can be evicted sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15338) Add a option to disable the data block cache for testing the performance of underlying file system
[ https://issues.apache.org/jira/browse/HBASE-15338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186670#comment-15186670 ] Hudson commented on HBASE-15338: FAILURE: Integrated in HBase-Trunk_matrix #766 (See [https://builds.apache.org/job/HBase-Trunk_matrix/766/]) HBASE-15420: TestCacheConfig failed after HBASE-15338 (liushaohui: rev 7e4e8dc3e9e0edb287fe27e488736c906b1d0ff5) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java > Add a option to disable the data block cache for testing the performance of > underlying file system > -- > > Key: HBASE-15338 > URL: https://issues.apache.org/jira/browse/HBASE-15338 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15338-trunk-v1.diff, HBASE-15338-trunk-v2.diff, > HBASE-15338-trunk-v3.diff, HBASE-15338-trunk-v4.diff, > HBASE-15338-trunk-v5.diff, HBASE-15338-trunk-v6.diff, > HBASE-15338-trunk-v7.diff > > > When testing and comparing the performance of different file systems(HDFS, > Azure blob storage, AWS S3 and so on) for HBase, it's better to avoid the > affect of the HBase BlockCache and get the actually random read latency when > data block is read from underlying file system. (Usually, the index block and > meta block should be cached in memory in the testing). > So we add a option in CacheConfig to disable the data block cache. > Suggestions are welcomed~ Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186669#comment-15186669 ] Hudson commented on HBASE-15422: FAILURE: Integrated in HBase-Trunk_matrix #766 (See [https://builds.apache.org/job/HBase-Trunk_matrix/766/]) HBASE-15422 Procedure v2 - Avoid double yield (matteo.bertozzi: rev 648cc5e8233f548b737a6d38683b66148fcf6e7f) * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestYieldProcedures.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186668#comment-15186668 ] Hudson commented on HBASE-15420: FAILURE: Integrated in HBase-Trunk_matrix #766 (See [https://builds.apache.org/job/HBase-Trunk_matrix/766/]) HBASE-15420: TestCacheConfig failed after HBASE-15338 (liushaohui: rev 7e4e8dc3e9e0edb287fe27e488736c906b1d0ff5) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15271) Spark Bulk Load: Need to write HFiles to tmp location then rename to protect from Spark Executor Failures
[ https://issues.apache.org/jira/browse/HBASE-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186671#comment-15186671 ] Hudson commented on HBASE-15271: FAILURE: Integrated in HBase-Trunk_matrix #766 (See [https://builds.apache.org/job/HBase-Trunk_matrix/766/]) HBASE-15271 Spark bulk load should write to temporary location and then (busbey: rev b29ce7f1144e31bce9d9862d25324029def8dbad) * hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala > Spark Bulk Load: Need to write HFiles to tmp location then rename to protect > from Spark Executor Failures > - > > Key: HBASE-15271 > URL: https://issues.apache.org/jira/browse/HBASE-15271 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 2.0.0 > > Attachments: HBASE-15271.1.patch, HBASE-15271.2.patch, > HBASE-15271.3.patch, HBASE-15271.4.patch > > > With the current code if an executor failure before the HFile is close it > will cause problems. This jira will have the files first write out to a file > that starts with an underscore. Then when the HFile is complete it will be > renamed and the underscore will be removed. > The underscore is important because the load bulk functionality will skip > files with an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15113) Procedure v2 - Speedup eviction of sys operation results
[ https://issues.apache.org/jira/browse/HBASE-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186663#comment-15186663 ] Hudson commented on HBASE-15113: SUCCESS: Integrated in HBase-1.3-IT #545 (See [https://builds.apache.org/job/HBase-1.3-IT/545/]) HBASE-15113 Procedure v2 - Speedup eviction of sys operation results (matteo.bertozzi: rev 46bf3c32750fd8e03271dd5d8d99918149b928f7) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java > Procedure v2 - Speedup eviction of sys operation results > > > Key: HBASE-15113 > URL: https://issues.apache.org/jira/browse/HBASE-15113 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.2.0, 1.1.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15113-v0.patch, HBASE-15113-v1.patch, > HBASE-15113-v2.patch, HBASE-15113-v3.patch > > > Some system operations like create system tables or namespaces or the server > crash handler. can be evicted sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186664#comment-15186664 ] Hudson commented on HBASE-15422: SUCCESS: Integrated in HBase-1.3-IT #545 (See [https://builds.apache.org/job/HBase-1.3-IT/545/]) HBASE-15422 Procedure v2 - Avoid double yield (matteo.bertozzi: rev c60da6961227f3e78530a877f4a435eb70738b10) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestYieldProcedures.java > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186643#comment-15186643
]
Mikhail Antonov commented on HBASE-15406:
-
How many such tools we could have in mind, which might require cluster-side
recovery? Any lease recovery for the tool is probably going to be somewhat
bulky/boilerplate, so if we need /want that for more than one tool (do we?)
would be good to avoid doing that just for hbck.
Another way is to try to address it in a cheap way which covers most of
probable issues. May be as I noted above, we can address hitting Ctrl-C by
adding a signal handler (and making sure we intercept any exceptions inside
hbck itself at highest levels to do that recovery). Probability of machine
running scheduled hbck cron jobs getting killed or loosing network is low (we
just need to make sure hbck itself doesn't exit without exception handling).
Probability of admin aborting command and leaving without reading any notices
is higher imho.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389-v5.patch Addressing comments on rb. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.19 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389-v2.patch, HBASE-15389-v3.patch, HBASE-15389-v4.patch, > HBASE-15389-v5.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-15377:
--
Attachment: HBASE-15377_v2.patch
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15377.patch, HBASE-15377_v1.patch,
> HBASE-15377_v2.patch
>
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186635#comment-15186635 ] Phil Yang commented on HBASE-15398: --- I find if I add a client-side logic to judge if joinedHeap will have families, it will break TestScanEarlyTermination and TestVisibilityLabelsWithACL, because we need getTableDescriptor at client side and may have no permission. We may can only throw an exception at server when user setAllowPartailResults && joinedHeap!=null? And for same reason, I can only sort each time in createCompleteResult. Any ideas? > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15425) Failing to write bulk load event marker in the WAL is ignored
[
https://issues.apache.org/jira/browse/HBASE-15425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186631#comment-15186631
]
Ashish Singhi commented on HBASE-15425:
---
bq. What is the specific reason for the WAL markers to be igonored?
I think it is a bug. We should fix it.
{quote}
Suppose a region open marker is ignored due to exception what happens on Log
replay?
How is the flush marker getting reused from the WAL?
{quote}
I need to check code fully before I comment anything about them. But these can
be done later also I think, not as part of this jira.
> Failing to write bulk load event marker in the WAL is ignored
> -
>
> Key: HBASE-15425
> URL: https://issues.apache.org/jira/browse/HBASE-15425
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>
> During LoadIncrementalHFiles process if we fail to write the bulk load event
> marker in the WAL, it is ignored. So this will lead to data mismatch issue in
> source and peer cluster in case of bulk loaded data replication scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15113) Procedure v2 - Speedup eviction of sys operation results
[ https://issues.apache.org/jira/browse/HBASE-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186627#comment-15186627 ] Hudson commented on HBASE-15113: SUCCESS: Integrated in HBase-1.3 #594 (See [https://builds.apache.org/job/HBase-1.3/594/]) HBASE-15113 Procedure v2 - Speedup eviction of sys operation results (matteo.bertozzi: rev 65651b50019c38df6b161ec1a17b5b06f71d97ae) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java > Procedure v2 - Speedup eviction of sys operation results > > > Key: HBASE-15113 > URL: https://issues.apache.org/jira/browse/HBASE-15113 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.2.0, 1.1.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15113-v0.patch, HBASE-15113-v1.patch, > HBASE-15113-v2.patch, HBASE-15113-v3.patch > > > Some system operations like create system tables or namespaces or the server > crash handler. can be evicted sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186628#comment-15186628 ] Hudson commented on HBASE-15422: SUCCESS: Integrated in HBase-1.3 #594 (See [https://builds.apache.org/job/HBase-1.3/594/]) HBASE-15422 Procedure v2 - Avoid double yield (matteo.bertozzi: rev 84aceed3b32e71f38125e1aaa8bb655e5969041a) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestYieldProcedures.java > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186625#comment-15186625 ] stack commented on HBASE-15180: --- The horrid names are fine by me. They are explicit and for internal use only. Otherwise patch looks good. Have you looked at moving what is common in IPCUtil up into hbase-common and then shutting down the access on IPCUtil so only accessible by ipc client? Can move its guts to some codec util up in hbase common and the server-side could use that. Is that a load of work? Is it doable? Ok to file a separate issue to do this. > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch, HBASE-15180_V6.patch, HBASE-15180_V7.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186624#comment-15186624 ] Anoop Sam John commented on HBASE-15325: Ya there will be only client side change - When a batch limit is NOT set on the scan, we will see whether the last fetched Result is a partial one. If so we will start with that row in the next scanner. If not the start row of the new scanner will be next row(as in current code). Also we will skip the cells what we already got. Or even we can clear the Results on this row what we have now - When a batch is set on scan, we will treat always that we are at middle of the row. And the new scanner will be having a start of row = this last row. Only exception is a short circuit (to avoid fetching one more row) where we will make next row as new scanner start row, when the last returned batch on this row is NOT partial but has lesser #cells than the batch size. Here we are sure that this is the last possible batch. > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15325-test.txt, HBASE-15325-v1.txt, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186620#comment-15186620
]
Amal Joshy commented on HBASE-15314:
{quote}
The point and need here is that as when you are filling up try to parallelize
the files instead of filling it up one by one. So that initial 20G itself
should be distributed among the four files. Is that so?
{quote}
Yes, that is what we are trying to achieve. But the problem is that the
BucketAllocator finds us an offset to store a block and this offset will be
generated in such a way that all the blocks will be mapped into the same file
until it is filled completely. We cannot obtain any random writes due to this.
> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more
> than one SSD in it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186614#comment-15186614
]
Heng Chen commented on HBASE-15377:
---
Fine. All we need is some kind uniform name. Because
Mutate,Delete,Increment,Append,Replay also have same problem.
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15377.patch, HBASE-15377_v1.patch
>
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186595#comment-15186595
]
stack commented on HBASE-15398:
---
The storeHeap/joinedHeap came in here:
{code}
commit 39954ddcd4b6e825012f394072d38a92b2171fa1
Author: Zhihong Yu
Date: Wed Jan 9 21:37:35 2013 +
HBASE-5416 Improve performance of scans with some kind of filters (Max
Lapan and Sergey)
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1431103
13f79535-47bb-0310-9956-ffa450edef68
{code}
A.K.A "Filter on one CF and if a match, then load and return full row (WAS:
Improve performance of scans with some kind of filters)"
bq. We scan storeHeap first, then joinedHeap, and merge the results and sort
and return to client.
Everywhere we merge sort except in this case? And because of this one case, we
have to add a sort to Result (previous we relied on Cells coming up out of the
server in order IIRC)?
So, if the SCVF filter is in place, we won't do partials? And we'll not do
partials silently? i.e. we'll just not go the partial route if SCVF is in place?
Good find [~yangzhe1991]
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1
[ https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186594#comment-15186594 ] Yu Li commented on HBASE-15120: --- Checking the latest UT, TestSplitTransactionOnCluster passed in both jdk7 and jdk8. Since changes in the patch are limited within TestSplitTransactionOnCluster, none of the failed case is caused by this patch, nor does the javadoc warning. I think we are good to get this one in sir [~ndimiduk] > Backport HBASE-14883 to branch-1.1 > -- > > Key: HBASE-15120 > URL: https://issues.apache.org/jira/browse/HBASE-15120 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 1.1.4 > > Attachments: HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch, HBASE-15120.branch-1.1.patch, > HBASE-15120.branch-1.1.patch > > > When checking branch-1.1 UT in HBASE-13590, found > TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. > The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186584#comment-15186584 ] ramkrishna.s.vasudevan commented on HBASE-15314: bq.If your bucketcache usedsize is only 20GB you get the performance of a single physical disk with the proposed patch. If you use all files in parallel you get the performance of all 4 disks. I am trying learn more here. So here the point is as and when the cache is filling up only then the usedSize will be growing and reaching upto 100GB (which is the total size of bucketCache). The point and need here is that as when you are filling up try to parallelize the files instead of filling it up one by one. So that initial 20G itself should be distributed among the four files. Is that so? > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15113) Procedure v2 - Speedup eviction of sys operation results
[
https://issues.apache.org/jira/browse/HBASE-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186581#comment-15186581
]
Hadoop QA commented on HBASE-15113:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 29s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 28s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 3s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 29s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 51s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 52s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 28s
{color} | {color:green} hbase-procedure in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 32s
{color} | {color:green} hbase-procedure in the patch passed with JDK v1.8.0_74.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186575#comment-15186575
]
Amal Joshy commented on HBASE-15314:
{quote}
this is more like fileEndingOffset?
{quote}
Exactly, I'll rename it in the next patch. Thanks [~ram_krish]
> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more
> than one SSD in it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186567#comment-15186567
]
stack commented on HBASE-15406:
---
This issue is 'critical'?
bq. I really don't like one thread to do cleanup in master just for hbck.
Me either.
hbck is an admin tool. Why can't the admin tool, for certain operations,
complain loudly that if it exists prematurely, state may be left set in the
master and explain how to explore and rectify.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186565#comment-15186565
]
Heng Chen commented on HBASE-15406:
---
Oh, sorry. Just see it.
{quote}
If so, changing the switch from hbck would involve adjusting value of current
znode and adding ephemeral znode at the same time. The values stored in current
znode would need to expand beyond true / false because we want to distinguish
the scenario where hbck (or some other tool) turns off split / merge for the
duration of the operation.
{quote}
It is really a problem. Maybe we should add another interface to distinguish
the operation from tool.
{quote}
Would the ephemeral znode be child of split / merge znode?
{quote}
I think so.
{quote}
What if hbck tries to turn off split while come other tool tries to turn on
split at the same time ?
{quote}
I think the lease is exclusive, when the split / merge lease was token by one
tool, the other tools could not acquire it.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186553#comment-15186553
]
JunHo Cho commented on HBASE-15430:
---
thank you. I will add test case to this patch.
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Priority: Critical
> Attachments: hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186552#comment-15186552 ] ramkrishna.s.vasudevan commented on HBASE-15314: bq.fileEndSizes this is more like fileEndingOffset? bq. atleast once all the files are filled, we can guarantee random reads utilizing all the disks. Yes this is true. Reads are random always. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186550#comment-15186550
]
Matteo Bertozzi commented on HBASE-15430:
-
you have two ways. or you generate the manifest protobuf directly with a for
loop.
or you use SnapshotManifest.addRegion() with a mocked region in a for loop.
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Priority: Critical
> Attachments: hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnaps
[jira] [Commented] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186547#comment-15186547
]
JunHo Cho commented on HBASE-15430:
---
i think that it is not easy to create very big manifest file (over 64MB). it
there any way to make manifest easily? if not, I attached big file for test.
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Priority: Critical
> Attachments: hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapsho
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186544#comment-15186544 ] stack commented on HBASE-15314: --- Thanks [~Amal Joshy] bq. But it is not possible to obtain parallel writes with the current BucketAllocator implementation. Why is this please? A bucket can't have an associated file? That'd be breaking change? > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186534#comment-15186534
]
Ted Yu commented on HBASE-15406:
That was what I talked about earlier:
https://issues.apache.org/jira/browse/HBASE-15406?focusedCommentId=15183264&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15183264
Would the ephemeral znode be child of split / merge znode ?
What if hbck tries to turn off split while come other tool tries to turn on
split at the same time ? This can be handled by using zookeeper multi operation
to avoid inconsistent state.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186526#comment-15186526 ] Phil Yang commented on HBASE-15398: --- It seems that if we put all cf into joinedHeap, we will not scan out anything. Or rather if storeHeap doesn't contain a row, scanner can not get this row even if joinedHeap has this row. Wondering is this true and expected? > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186508#comment-15186508
]
Heng Chen edited comment on HBASE-15406 at 3/9/16 5:06 AM:
---
This is my plan.
When some tool (for example, hbck) running, it will create an ephemeral znode.
We record the tool's state in this znode. Meaning while master will be
notified that there is some kind tool's ephemeral znode was created (we can
call it the tool's lease) and master will watch it.
When the tool abort, znode will be deleted and master will do rollback
according to the tool's type and state. Currently we can implement the first
tool logic "hbck".
wdyt? [~tedyu]
was (Author: chenheng):
This is my plan.
When some tool (for example, hbck) running, it will create an ephemeral znode.
We record the tool's state in this znode. Meaning while master will be noticed
that there is some kind tool's ephemeral znode was created (we can call it the
tool's lease) and master will watch it.
When the tool abort, znode will be deleted and master will do rollback
according to the tool's type and state. Currently we can implement the first
tool logic "hbck".
wdyt? [~tedyu]
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186508#comment-15186508
]
Heng Chen commented on HBASE-15406:
---
This is my plan.
When some tool (for example, hbck) running, it will create an ephemeral znode.
We record the tool's state in this znode. Meaning while master will be noticed
that there is some kind tool's ephemeral znode was created (we can call it the
tool's lease) and master will watch it.
When the tool abort, znode will be deleted and master will do rollback
according to the tool's type and state. Currently we can implement the first
tool logic "hbck".
wdyt? [~tedyu]
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[
https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186502#comment-15186502
]
ramkrishna.s.vasudevan commented on HBASE-15398:
To start with (I may be wrong here reading this code now)
{code}
if (this.filter == null || !scan.doLoadColumnFamiliesOnDemand()
|| this.filter.isFamilyEssential(entry.getKey())) {
scanners.add(scanner);
} else {
joinedScanners.add(scanner);
}
{code}
If there is a SCVF configured and scan says setLoadColumnFamiliesOnDemand(true)
- we will only have joinedHeap active on the RegionScanner? Then storeHeap is
guarded enough not to fetch results from it? And will the scan continue with
joinedHeap?
> Cells loss or disorder when using family essential filter and partial
> scanning protocol
> ---
>
> Key: HBASE-15398
> URL: https://issues.apache.org/jira/browse/HBASE-15398
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: 15398-test.txt, HBASE-15398.v1.txt
>
>
> In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have
> a filter and it doesn't apply to all cf, the stores whose families needn't be
> filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap,
> and merge the results and sort and return to client. We need sort because the
> order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap.
> However, after HBASE-11544 we may transfer partial results when we get
> SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf
> first because it is in storeHeap and then a smaller cf because it is in
> joinedHeap. Server won't hold all cells in a row and client doesn't have a
> sorting logic. The order of cf in Result for user is wrong.
> And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last
> cell of a row in storeHeap, we will break scanning in RegionScannerImpl and
> in populateResult we will change the state to SIZE_LIMIT_REACHED because next
> peeked cell is next row. But this is only the last cell of one and we have
> two... And SIZE_LIMIT_REACHED means this Result is not partial (by
> ScannerContext.partialResultFormed), client will see it and merge them and
> return to user with losing data of joinedHeap. On next scan we will read next
> row of storeHeap and joinedHeap is forgotten and never be read...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15412) Add average region size metric
[ https://issues.apache.org/jira/browse/HBASE-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alicia Ying Shu updated HBASE-15412: Status: Patch Available (was: Open) > Add average region size metric > -- > > Key: HBASE-15412 > URL: https://issues.apache.org/jira/browse/HBASE-15412 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alicia Ying Shu > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15412.patch > > > We have several metrics related to region store file size, num regions, etc > per regionserver, but we do not have a single metric to track the average > region size per regionserver. > Avg region size is important to look at for deciding on the split policy, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15412) Add average region size metric
[ https://issues.apache.org/jira/browse/HBASE-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186500#comment-15186500 ] Alicia Ying Shu commented on HBASE-15412: - Uploaded the patch. Thanks [~enis] for reviewing. > Add average region size metric > -- > > Key: HBASE-15412 > URL: https://issues.apache.org/jira/browse/HBASE-15412 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alicia Ying Shu > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15412.patch > > > We have several metrics related to region store file size, num regions, etc > per regionserver, but we do not have a single metric to track the average > region size per regionserver. > Avg region size is important to look at for deciding on the split policy, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15412) Add average region size metric
[ https://issues.apache.org/jira/browse/HBASE-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alicia Ying Shu updated HBASE-15412: Attachment: HBASE-15412.patch > Add average region size metric > -- > > Key: HBASE-15412 > URL: https://issues.apache.org/jira/browse/HBASE-15412 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alicia Ying Shu > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15412.patch > > > We have several metrics related to region store file size, num regions, etc > per regionserver, but we do not have a single metric to track the average > region size per regionserver. > Avg region size is important to look at for deciding on the split policy, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186497#comment-15186497 ] Amal Joshy commented on HBASE-15314: I understand your concern [~danielpol]. But it is not possible to obtain parallel writes with the current BucketAllocator implementation. Either we need to change the current implementation or plugin a different BucketAllocator only for FileIOEngine. With this patch, atleast once all the files are filled, we can guarantee random reads utilizing all the disks. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15425) Failing to write bulk load event marker in the WAL is ignored
[ https://issues.apache.org/jira/browse/HBASE-15425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186483#comment-15186483 ] ramkrishna.s.vasudevan commented on HBASE-15425: What is the specific reason for the WAL markers to be igonored? Suppose a region open marker is ignored due to exception what happens on Log replay? How is the flush marker getting reused from the WAL? I have not seen this code fully to know the full behaviour. > Failing to write bulk load event marker in the WAL is ignored > - > > Key: HBASE-15425 > URL: https://issues.apache.org/jira/browse/HBASE-15425 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > > During LoadIncrementalHFiles process if we fail to write the bulk load event > marker in the WAL, it is ignored. So this will lead to data mismatch issue in > source and peer cluster in case of bulk loaded data replication scenario. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186478#comment-15186478 ] ramkrishna.s.vasudevan commented on HBASE-15325: So the final conclusion is that we are not going to treat BATCH_LIMIT_REACH as partial on the server side because the result returned to the cleint will have the isPartial() set on it. And hence only the client side change is going to be applied as in the current patch and in addition to it if in setBatch we get lesser results we will again fetch from that lastResult of that batch, right? > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15325-test.txt, HBASE-15325-v1.txt, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15431) A bunch of methods are hot and too big to be inlined
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186466#comment-15186466 ] ramkrishna.s.vasudevan commented on HBASE-15431: This is 0.98 I belive? I think in trunk bq.KeyValue::createByteArray Use of this is avoided as much as possible. If still there then an area that could be solved. > A bunch of methods are hot and too big to be inlined > > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). > In all cases I primed the JVM to make sure the JVM gets a chance to profile > the methods and decide whether they're hot or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15422) Procedure v2 - Avoid double yield
[ https://issues.apache.org/jira/browse/HBASE-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15422: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - Avoid double yield > - > > Key: HBASE-15422 > URL: https://issues.apache.org/jira/browse/HBASE-15422 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15422-v0.patch > > > ServerCrashProcedure is using a combination of > isYieldBeforeExecuteFromState() and ProcedureYieldException, which may end up > in yielding twice. (ServerCrashProcedure is the only user of yield) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15113) Procedure v2 - Speedup eviction of sys operation results
[ https://issues.apache.org/jira/browse/HBASE-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15113: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - Speedup eviction of sys operation results > > > Key: HBASE-15113 > URL: https://issues.apache.org/jira/browse/HBASE-15113 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.2.0, 1.1.2 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1 > > Attachments: HBASE-15113-v0.patch, HBASE-15113-v1.patch, > HBASE-15113-v2.patch, HBASE-15113-v3.patch > > > Some system operations like create system tables or namespaces or the server > crash handler. can be evicted sooner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15364) Fix unescaped < characters in Javadoc
[
https://issues.apache.org/jira/browse/HBASE-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186435#comment-15186435
]
Hudson commented on HBASE-15364:
FAILURE: Integrated in HBase-Trunk_matrix #765 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/765/])
HBASE-15364 Fix unescaped < and > characters in Javadoc (mstanleyjones: rev
14217cef247f5ab0f052b54fc569565137e566e0)
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> Fix unescaped < characters in Javadoc
> -
>
> Key: HBASE-15364
> URL: https://issues.apache.org/jira/browse/HBASE-15364
> Project: HBase
> Issue Type: Bug
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Gabor Liptak
> Labels: newbie
> Fix For: 2.0.0
>
> Attachments: HBASE-15364.1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/28/artifact/link_report/index.html:
>
> {code}
> host: hbase.apache.org
> date: Mon, 29-Feb-2016 12:06:21 (local)
> Linklint version: 2.3.5_ingo_020
> #
> # warn2 warnings (cross referenced)
> #
> unquoted "<" in <0.90.5, <0.90.5, <
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> unquoted "<" in <0.92.0) a master
> res
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186410#comment-15186410 ] Anoop Sam John commented on HBASE-15325: bq.if setBatch or last result returned to user is partial we need rescan the row of last result, right? Exactly. bq. if we get a result whose isPartial==false and size < scan.getBatch, it must be the last result or this row, we can return it to user more quickly, right? Excellent.. Ya that is a short circuit possible here with out any addition of new state to Result.. Yes we can do that.. Pls add enough comments around these kind of tricky logic why we can do the short circuit. Good on you. > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15325-test.txt, HBASE-15325-v1.txt, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186406#comment-15186406 ] Anoop Sam John commented on HBASE-15180: Ya to make sure the client uses only which it supposed to, we can rename method at least. Any better name suggestion boss [~saint@gmail.com]? The best way would have been that, the codec instance know where am I. Whether it is in client context or server. We need to set it while constructing Codec instance. But ya at least a method name change we can do now. I will fix ur suggestion abt explicit javadoc statement on commit. Thanks for the review. Once Stack also good abt the patch, will commit. > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch, HBASE-15180_V6.patch, HBASE-15180_V7.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15378) Scanner can not handle a heartbeat message with no results
[ https://issues.apache.org/jira/browse/HBASE-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186400#comment-15186400 ] Phil Yang commented on HBASE-15378: --- Done, thanks. > Scanner can not handle a heartbeat message with no results > -- > > Key: HBASE-15378 > URL: https://issues.apache.org/jira/browse/HBASE-15378 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: HBASE-15378-v1.txt, HBASE-15378-v2.txt, > HBASE-15378-v3.txt, HBASE-15378-v4.patch, HBASE-15378-v5.patch > > > When a RS scanner get a TIME_LIMIT_REACHED_MID_ROW state, they will stop > scanning and send back what it has read to client and mark the message as a > heartbeat message. If there is no cell has been read, it will be an empty > response. > However, ClientScanner only handles the situation that the client gets an > empty heartbeat and its cache is not empty. If the cache is empty too, it > will be regarded as end-of-region and open a new scanner for next region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15378) Scanner can not handle a heartbeat message with no results
[ https://issues.apache.org/jira/browse/HBASE-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15378: -- Attachment: HBASE-15378-v5.patch > Scanner can not handle a heartbeat message with no results > -- > > Key: HBASE-15378 > URL: https://issues.apache.org/jira/browse/HBASE-15378 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: HBASE-15378-v1.txt, HBASE-15378-v2.txt, > HBASE-15378-v3.txt, HBASE-15378-v4.patch, HBASE-15378-v5.patch > > > When a RS scanner get a TIME_LIMIT_REACHED_MID_ROW state, they will stop > scanning and send back what it has read to client and mark the message as a > heartbeat message. If there is no cell has been read, it will be an empty > response. > However, ClientScanner only handles the situation that the client gets an > empty heartbeat and its cache is not empty. If the cache is empty too, it > will be regarded as end-of-region and open a new scanner for next region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15417) Calls to ObserverContext#bypass in a region observer's prePut method are inconsistent
[
https://issues.apache.org/jira/browse/HBASE-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186396#comment-15186396
]
Anoop Sam John commented on HBASE-15417:
This is not a test issue.. This is a bug only IMO.
We call batchMutate with some puts. 1st we will call prePut() on all of them
one after the other. When for any put, the CP calls bypass, we will mark that
Mutation status as success. Then we will call doMiniBatchMutate().. The actual
mutation happens in 1 or more mini batches as we obtain rowlocks.
Then we try to obtain rowlocks in doMiniBatchMutate() iff the status of that
mutation is not done. But the pre hook made it to success. So no need to obtain
row lock and there is 0 puts to write. Then we have this code in
doMiniBatchMutate
{code}
if (numReadyToWrite <= 0) {
return 0L;
}
{code}
The post hook is after the actual write on memstore and WAL in
doMiniBatchMutate.
So when all puts handled by pre hook and bypassed, there is no call to post
hook.
But when at least one of the put is not done by pre hook and actual write on
memstore happens in doMiniBatchMutate, we will call post hook on ALL puts which
appeared in THAT mini batch. So it is not even all the puts.. If all puts can
happen in one mini batch, yes on all we will call post hook.
The problem in when we call post hook, we know the status of the op and success
or not and dont know whether it is set by pre hook call and bypass or by the
actual put on memstore. If we can distinguish that and decide to call post
hook based on that, we can solve this
Even the pre and post hook for batch mutate (pre/postBatchMutate) also to be
consistent with pre/postPut. Ie. when prePut bypass one mutation, that
mutation should not be part of mutations that we pass to pre/postBatchMutate
hooks.
If u can work on a patch based on the above we will reviews.. If u are busy ,
let me know, I can give a patch :-)
Thanks for the find..
> Calls to ObserverContext#bypass in a region observer's prePut method are
> inconsistent
> -
>
> Key: HBASE-15417
> URL: https://issues.apache.org/jira/browse/HBASE-15417
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Harry Harpham
>Priority: Minor
>
> Calling ctx.bypass(), where ctx is the ObserverContext object passed in to
> the region observer's prePut method, results in some inconsistent behavior.
> If every other put in the batch is also bypassed, the region observer sees
> none of these in its postPut method. If there is at least one other put
> which is not bypassed, the region observer sees all of the puts in the batch
> _including those which were bypassed_.
> The end result is that, after bypassing a put, that put may or may not end up
> in the region observer's postPut method. This behavior is dependent solely
> on which other puts the bypassed put is batched together with.
> I tried to find existing tickets for this issue, but was unable to.
> Apologies if I missed something. The closest issues I could find were
> HBASE-4331 and HBASE-11503, but those didn't seem to quite hit it.
> Additionally, I threw together a quick demonstration of this issue:
> https://github.com/hwh33/bypass-inconsistency-demo. You can run that demo in
> memory using the testing utility or against a running cluster. I actually
> haven't had time to test it against a cluster though, so you may encounter
> bugs if running in that mode (but hopefully not!).
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15417) Calls to ObserverContext#bypass in a region observer's prePut method are inconsistent
[
https://issues.apache.org/jira/browse/HBASE-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186396#comment-15186396
]
Anoop Sam John edited comment on HBASE-15417 at 3/9/16 3:06 AM:
This is not a test issue.. This is a bug only IMO.
We call batchMutate with some puts. 1st we will call prePut() on all of them
one after the other. When for any put, the CP calls bypass, we will mark that
Mutation status as success. Then we will call doMiniBatchMutate().. The actual
mutation happens in 1 or more mini batches as we obtain rowlocks.
Then we try to obtain rowlocks in doMiniBatchMutate() iff the status of that
mutation is not done. But the pre hook made it to success. So no need to obtain
row lock and there is 0 puts to write. Then we have this code in
doMiniBatchMutate
{code}
if (numReadyToWrite <= 0) {
return 0L;
}
{code}
The post hook is after the actual write on memstore and WAL in
doMiniBatchMutate.
So when all puts handled by pre hook and bypassed, there is no call to post
hook.
But when at least one of the put is not done by pre hook and actual write on
memstore happens in doMiniBatchMutate, we will call post hook on ALL puts which
appeared in THAT mini batch. So it is not even all the puts.. If all puts can
happen in one mini batch, yes on all we will call post hook.
The problem in when we call post hook, we know the status of the op and success
or not and dont know whether it is set by pre hook call and bypass or by the
actual put on memstore. If we can distinguish that and decide to call post
hook based on that, we can solve this
Even the pre and post hook for batch mutate (pre/postBatchMutate) also to be
consistent with pre/postPut. Ie. when prePut bypass one mutation, that
mutation should not be part of mutations that we pass to pre/postBatchMutate
hooks.
If u can work on a patch based on the above we will review.. If u are busy ,
let me know, I can give a patch :-)
Thanks for the find..
was (Author: anoop.hbase):
This is not a test issue.. This is a bug only IMO.
We call batchMutate with some puts. 1st we will call prePut() on all of them
one after the other. When for any put, the CP calls bypass, we will mark that
Mutation status as success. Then we will call doMiniBatchMutate().. The actual
mutation happens in 1 or more mini batches as we obtain rowlocks.
Then we try to obtain rowlocks in doMiniBatchMutate() iff the status of that
mutation is not done. But the pre hook made it to success. So no need to obtain
row lock and there is 0 puts to write. Then we have this code in
doMiniBatchMutate
{code}
if (numReadyToWrite <= 0) {
return 0L;
}
{code}
The post hook is after the actual write on memstore and WAL in
doMiniBatchMutate.
So when all puts handled by pre hook and bypassed, there is no call to post
hook.
But when at least one of the put is not done by pre hook and actual write on
memstore happens in doMiniBatchMutate, we will call post hook on ALL puts which
appeared in THAT mini batch. So it is not even all the puts.. If all puts can
happen in one mini batch, yes on all we will call post hook.
The problem in when we call post hook, we know the status of the op and success
or not and dont know whether it is set by pre hook call and bypass or by the
actual put on memstore. If we can distinguish that and decide to call post
hook based on that, we can solve this
Even the pre and post hook for batch mutate (pre/postBatchMutate) also to be
consistent with pre/postPut. Ie. when prePut bypass one mutation, that
mutation should not be part of mutations that we pass to pre/postBatchMutate
hooks.
If u can work on a patch based on the above we will reviews.. If u are busy ,
let me know, I can give a patch :-)
Thanks for the find..
> Calls to ObserverContext#bypass in a region observer's prePut method are
> inconsistent
> -
>
> Key: HBASE-15417
> URL: https://issues.apache.org/jira/browse/HBASE-15417
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Harry Harpham
>Priority: Minor
>
> Calling ctx.bypass(), where ctx is the ObserverContext object passed in to
> the region observer's prePut method, results in some inconsistent behavior.
> If every other put in the batch is also bypassed, the region observer sees
> none of these in its postPut method. If there is at least one other put
> which is not bypassed, the region observer sees all of the puts in the batch
> _including those which were bypassed_.
> The end result is that, after bypassing a put, that put may or may not end up
> in the region observer's postPut method. This behavior is dependent solely
> on which other puts the bypassed put is batched toge
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186393#comment-15186393 ] Phil Yang commented on HBASE-15325: --- Oh I get your point now. Sounds reasonable. So we need also judge whether user setBatch or not when we open a new scanner, if setBatch or last result returned to user is partial we need rescan the row of last result, right? And this solution has benefit I think, if we get a result whose isPartial==false and size < scan.getBatch, it must be the last result or this row, we can return it to user more quickly, right? > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15325-test.txt, HBASE-15325-v1.txt, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15378) Scanner can not handle a heartbeat message with no results
[
https://issues.apache.org/jira/browse/HBASE-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186390#comment-15186390
]
Ted Yu commented on HBASE-15378:
Please insert the following in the catch clause:
{code}
Thread.currentThread().interrupt();
{code}
> Scanner can not handle a heartbeat message with no results
> --
>
> Key: HBASE-15378
> URL: https://issues.apache.org/jira/browse/HBASE-15378
> Project: HBase
> Issue Type: Bug
> Components: dataloss, Scanners
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Critical
> Attachments: HBASE-15378-v1.txt, HBASE-15378-v2.txt,
> HBASE-15378-v3.txt, HBASE-15378-v4.patch
>
>
> When a RS scanner get a TIME_LIMIT_REACHED_MID_ROW state, they will stop
> scanning and send back what it has read to client and mark the message as a
> heartbeat message. If there is no cell has been read, it will be an empty
> response.
> However, ClientScanner only handles the situation that the client gets an
> empty heartbeat and its cache is not empty. If the cache is empty too, it
> will be regarded as end-of-region and open a new scanner for next region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186386#comment-15186386 ] Anoop Sam John commented on HBASE-15325: bq.And we need add BATCH_LIMIT_REACH to partialResultFormed() at server side. U mean treat batch limit reach case Results also as partial when send back from RS to client? We should not. Let us keep the meaning of partial as is.. If we want to clearly know whether a rows all batches are done or not (When region move case) let us clearly say it as a new state or so. But as I stated above, it is a rare case. So when a region move happens and if the batched scan in progress, let us assume the last fetched row is incomplete and read it again. (Just like how we will check whether last Result is partial or not).. Ya in a case where all batches of that row were fetched and next row's no batches are fetched and exactly at that time the region moved, we will do an unwanted row fetch. But what is the possibility of such a strict case? IMHO it is ok for that overhead. > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15325-test.txt, HBASE-15325-v1.txt, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[
https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186384#comment-15186384
]
Enis Soztutar commented on HBASE-15180:
---
The patch looks much cleaner than v4 I think.
Let's add a javadoc to this saying that the ownership / lifecycle of the passed
BB is transferred to the Decoder and Cells returned from the decoder, so that
the contract is explicit:
{code}
+ Decoder getDecoder(ByteBuffer buf);
{code}
Again, the contract for the {{IPCUtil. createCellScanner()}} used by client
side vs server side should be explicit. How do we even know that one is used
only by the client or server? We can declare it in the method name explicitly,
like {{createCellScannerReusingBuffers()}} (sorry for the horrible name).
Otherwise looks good. [~saint@gmail.com] give it a quick glance?
> Reduce garbage created while reading Cells from Codec Decoder
> -
>
> Key: HBASE-15180
> URL: https://issues.apache.org/jira/browse/HBASE-15180
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15180.patch, HBASE-15180_V2.patch,
> HBASE-15180_V4.patch, HBASE-15180_V6.patch, HBASE-15180_V7.patch
>
>
> In KeyValueDecoder#parseCell (Default Codec decoder) we use
> KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create
> a byte[] of length 4 and read the cell length and then an array of Cell's
> length and read in cell bytes into it and create a KV.
> Actually in server we read the reqs into a byte[] and CellScanner is created
> on top of a ByteArrayInputStream on top of this. By default in write path, we
> have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell
> bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that
> bytes. So there is no issue if we create Cells over the RPC read byte[]
> directly here in Decoder. No need for 2 byte[] creation and copy for every
> Cell in request.
> My plan is to make a Cell aware ByteArrayInputStream which can read Cells
> directly from it.
> Same Codec path is used in client side also. There better we can avoid this
> direct Cell create and continue to do the copy to smaller byte[]s path. Plan
> to introduce some thing like a CodecContext associated with every Codec
> instance which can say the server/client context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15325) ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
[ https://issues.apache.org/jira/browse/HBASE-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186363#comment-15186363 ] Phil Yang commented on HBASE-15325: --- Yes, the main work is at client side. And we need add BATCH_LIMIT_REACH to partialResultFormed() at server side. I'll draft a new patch based on our conclusion after HBASE-15378 pushed to each branch. They may conflict each other. > ResultScanner allowing partial result will miss the rest of the row if the > region is moved between two rpc requests > --- > > Key: HBASE-15325 > URL: https://issues.apache.org/jira/browse/HBASE-15325 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15325-test.txt, HBASE-15325-v1.txt, HBASE-15325-v2.txt, > HBASE-15325-v3.txt, HBASE-15325-v5.txt, HBASE-15325-v6.1.txt, > HBASE-15325-v6.2.txt, HBASE-15325-v6.3.txt, HBASE-15325-v6.4.txt, > HBASE-15325-v6.5.txt, HBASE-15325-v6.txt > > > HBASE-11544 allow scan rpc return partial of a row to reduce memory usage for > one rpc request. And client can setAllowPartial or setBatch to get several > cells in a row instead of the whole row. > However, the status of the scanner is saved on server and we need this to get > the next part if there is a partial result before. If we move the region to > another RS, client will get a NotServingRegionException and open a new > scanner to the new RS which will be regarded as a new scan from the end of > this row. So the rest cells of the row of last result will be missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14801) Enhance the Spark-HBase connector catalog with json format
[ https://issues.apache.org/jira/browse/HBASE-14801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186361#comment-15186361 ] Zhan Zhang commented on HBASE-14801: [~jmhsieh] Mind taking a final pass? > Enhance the Spark-HBase connector catalog with json format > -- > > Key: HBASE-14801 > URL: https://issues.apache.org/jira/browse/HBASE-14801 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-14801-1.patch, HBASE-14801-2.patch, > HBASE-14801-3.patch, HBASE-14801-4.patch, HBASE-14801-5.patch, > HBASE-14801-6.patch, HBASE-14801-7.patch, HBASE-14801-8.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15378) Scanner can not handle a heartbeat message with no results
[ https://issues.apache.org/jira/browse/HBASE-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-15378: -- Attachment: HBASE-15378-v4.patch remove printStackTrace and ignore it > Scanner can not handle a heartbeat message with no results > -- > > Key: HBASE-15378 > URL: https://issues.apache.org/jira/browse/HBASE-15378 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: HBASE-15378-v1.txt, HBASE-15378-v2.txt, > HBASE-15378-v3.txt, HBASE-15378-v4.patch > > > When a RS scanner get a TIME_LIMIT_REACHED_MID_ROW state, they will stop > scanning and send back what it has read to client and mark the message as a > heartbeat message. If there is no cell has been read, it will be an empty > response. > However, ClientScanner only handles the situation that the client gets an > empty heartbeat and its cache is not empty. If the cache is empty too, it > will be regarded as end-of-region and open a new scanner for next region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15420) TestCacheConfig failed after HBASE-15338
[ https://issues.apache.org/jira/browse/HBASE-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-15420: Resolution: Fixed Status: Resolved (was: Patch Available) > TestCacheConfig failed after HBASE-15338 > > > Key: HBASE-15420 > URL: https://issues.apache.org/jira/browse/HBASE-15420 > Project: HBase > Issue Type: Test > Components: test >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15420-v1.diff > > > TestCacheConfig failed after HBASE-15338. > Fix it in this issue~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
[ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186348#comment-15186348 ] Enis Soztutar commented on HBASE-15180: --- bq. Dropped branch-1 based fixed versions. Adding new method into Codec is fine there? Even if, we may change BB to new type in 2.0 later. So better we won't add any thing. The only Codec that is used by outside is the Phoenix index codec as far as I know. The codec encodes some more cells for the index corresponding to the IndexedKeyValue's. We can modify Phoenix to compile with HBase-2.0 when time comes. Other than that, Codec's are not user-level classes, they are/should be framework level only. > Reduce garbage created while reading Cells from Codec Decoder > - > > Key: HBASE-15180 > URL: https://issues.apache.org/jira/browse/HBASE-15180 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, > HBASE-15180_V4.patch, HBASE-15180_V6.patch, HBASE-15180_V7.patch > > > In KeyValueDecoder#parseCell (Default Codec decoder) we use > KeyValueUtil#iscreate to read cells from the InputStream. Here we 1st create > a byte[] of length 4 and read the cell length and then an array of Cell's > length and read in cell bytes into it and create a KV. > Actually in server we read the reqs into a byte[] and CellScanner is created > on top of a ByteArrayInputStream on top of this. By default in write path, we > have MSLAB usage ON. So while adding Cells to memstore, we will copy the Cell > bytes to MSLAB memory chunks (default 2 MB size) and recreate Cells over that > bytes. So there is no issue if we create Cells over the RPC read byte[] > directly here in Decoder. No need for 2 byte[] creation and copy for every > Cell in request. > My plan is to make a Cell aware ByteArrayInputStream which can read Cells > directly from it. > Same Codec path is used in client side also. There better we can avoid this > direct Cell create and continue to do the copy to smaller byte[]s path. Plan > to introduce some thing like a CodecContext associated with every Codec > instance which can say the server/client context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15431) A bunch of methods are hot and too big to be inlined
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15431: -- Description: I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining" and then looked for "hot method too big" log lines. I'll attach a log of those messages. I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods (as long as they're hot, but actually didn't see any improvement). In all cases I primed the JVM to make sure the JVM gets a chance to profile the methods and decide whether they're hot or not. was: I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining" and then looked for "hot method too big" log lines. I'll attach a log of those messages. I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods (as long as they're hot, but actually didn't see any improvement). > A bunch of methods are hot and too big to be inlined > > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). > In all cases I primed the JVM to make sure the JVM gets a chance to profile > the methods and decide whether they're hot or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15431) A bunch of methods are hot and too big to be inlined
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186345#comment-15186345 ] Lars Hofhansl commented on HBASE-15431: --- * KeyValue::createByteArray ... scary > A bunch of methods are hot and too big to be inlined > > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15431) A bunch of methods are hot and too big to be inlined
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15431: -- Summary: A bunch of methods are hot and too big to be inlined (was: A bunch of methods are hot and too big to be inline) > A bunch of methods are hot and too big to be inlined > > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15431) A bunch of methods are hot and too big to be inline
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186342#comment-15186342 ] Lars Hofhansl commented on HBASE-15431: --- The load I ran was a Phoenix count\(*) query. Note that almost all interesting methods using during scanning are _not_ inlined. * SQM.match * StoreScanner.next * HRegion$RegionScannerImpl::nextInternal * HFileReaderV2$ScannerV2::blockSeek * FastDiffDeltaEncoder$1::decode * etc As said, increasing -XX:FreqInlineSize did not improve things. Presumably because now too much stuff is being inlined. > A bunch of methods are hot and too big to be inline > --- > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15398) Cells loss or disorder when using family essential filter and partial scanning protocol
[ https://issues.apache.org/jira/browse/HBASE-15398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186339#comment-15186339 ] Phil Yang commented on HBASE-15398: --- Still working :) I refactor RegionSannerImpl.nextInternal so I must be careful and make sure all test can pass. > Cells loss or disorder when using family essential filter and partial > scanning protocol > --- > > Key: HBASE-15398 > URL: https://issues.apache.org/jira/browse/HBASE-15398 > Project: HBase > Issue Type: Bug > Components: dataloss, Scanners >Affects Versions: 1.2.0, 1.1.3 >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Critical > Attachments: 15398-test.txt, HBASE-15398.v1.txt > > > In RegionScannerImpl, we have two heaps, storeHeap and joinedHeap. If we have > a filter and it doesn't apply to all cf, the stores whose families needn't be > filtered will be in joinedHeap. We scan storeHeap first, then joinedHeap, > and merge the results and sort and return to client. We need sort because the > order of Cell is rowkey/cf/cq/ts and a smaller cf may be in the joinedHeap. > However, after HBASE-11544 we may transfer partial results when we get > SIZE_LIMIT_REACHED_MID_ROW or other similar states. We may return a larger cf > first because it is in storeHeap and then a smaller cf because it is in > joinedHeap. Server won't hold all cells in a row and client doesn't have a > sorting logic. The order of cf in Result for user is wrong. > And a more critical bug is, if we get a LIMIT_REACHED_MID_ROW on the last > cell of a row in storeHeap, we will break scanning in RegionScannerImpl and > in populateResult we will change the state to SIZE_LIMIT_REACHED because next > peeked cell is next row. But this is only the last cell of one and we have > two... And SIZE_LIMIT_REACHED means this Result is not partial (by > ScannerContext.partialResultFormed), client will see it and merge them and > return to user with losing data of joinedHeap. On next scan we will read next > row of storeHeap and joinedHeap is forgotten and never be read... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15431) A bunch of methods are hot and too big to be inline
[ https://issues.apache.org/jira/browse/HBASE-15431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-15431: -- Attachment: hotMethods.txt List of hot and big methods. > A bunch of methods are hot and too big to be inline > --- > > Key: HBASE-15431 > URL: https://issues.apache.org/jira/browse/HBASE-15431 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: hotMethods.txt > > > I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions > -XX:+PrintInlining" and then looked for "hot method too big" log lines. > I'll attach a log of those messages. > I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods > (as long as they're hot, but actually didn't see any improvement). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15377) Per-RS Get metric is time based, per-region metric is size-based
[
https://issues.apache.org/jira/browse/HBASE-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186333#comment-15186333
]
Enis Soztutar commented on HBASE-15377:
---
Thanks [~chenheng] for taking on this patch. It looks good, except that I think
we can keep "Get" (as opposed to GetTime) as the name of the metric for region
server and region level, and introduce "GetSize" as the new metric. I know that
for scans we did "ScanSize" and "ScanTime", but in this case, we can keep
compatibility for the regionserver-level metric name. wdyt?
> Per-RS Get metric is time based, per-region metric is size-based
>
>
> Key: HBASE-15377
> URL: https://issues.apache.org/jira/browse/HBASE-15377
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15377.patch, HBASE-15377_v1.patch
>
>
> We have metrics for Get operations at the region server level and region
> level.
> {code}
>"Get_num_ops" : 4837505,
> "Get_min" : 0,
> "Get_max" : 296,
> "Get_mean" : 0.2934618155433431,
> "Get_median" : 0.0,
> "Get_75th_percentile" : 0.0,
> "Get_95th_percentile" : 1.0,
> "Get_99th_percentile" : 1.0,
> {code}
> and
> {code}
>"Namespace_hbase_table_meta_region_1588230740_metric_get_num_ops" : 103,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_min" : 450,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_max" : 470,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_mean" :
> 450.19417475728153,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_median" : 460.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_75th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_95th_percentile"
> : 470.0,
> "Namespace_hbase_table_meta_region_1588230740_metric_get_99th_percentile"
> : 470.0,
> {code}
> The problem is that the report values for the region server shows the
> latency, versus the reported values for the region shows the response sizes.
> There is no way of telling this without reading the source code.
> I think we should deprecate response size histograms in favor of latency
> histograms.
> See also HBASE-15376.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15431) A bunch of methods are hot and too big to be inline
Lars Hofhansl created HBASE-15431: - Summary: A bunch of methods are hot and too big to be inline Key: HBASE-15431 URL: https://issues.apache.org/jira/browse/HBASE-15431 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl I ran HBase with "-XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining" and then looked for "hot method too big" log lines. I'll attach a log of those messages. I tried to increase -XX:FreqInlineSize to 1010 to inline all these methods (as long as they're hot, but actually didn't see any improvement). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15430) Failed taking snapshot - Manifest proto-message too large
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15430:
Summary: Failed taking snapshot - Manifest proto-message too large (was:
Failed taking snapshot )
> Failed taking snapshot - Manifest proto-message too large
> -
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Priority: Critical
> Attachments: hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15430) Failed taking snapshot
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186319#comment-15186319
]
Matteo Bertozzi commented on HBASE-15430:
-
can you add a test case?
it should be easy to create a manifest and then just open it
> Failed taking snapshot
> ---
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Priority: Critical
> Attachments: hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15423) Fix integration issue came due HBASE-15216 from main to 0.98.
[
https://issues.apache.org/jira/browse/HBASE-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15423:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
+1
Committed to 0.98
> Fix integration issue came due HBASE-15216 from main to 0.98.
> --
>
> Key: HBASE-15423
> URL: https://issues.apache.org/jira/browse/HBASE-15423
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Minor
> Fix For: 0.98.18
>
> Attachments: HBASE-15423-0.98.patch
>
>
> https://issues.apache.org/jira/browse/HBASE-15216
> Sequence of call got reverse while merging. In code we are first doing the
> secure login ( which needs keytab/pricipal in config ) and then doing param
> parsing. Call should be other way round
> {code}
> // loading the generic options to conf
> new GenericOptionsParser(conf, args);
>
> AuthUtil.launchAuthChore(conf);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15271) Spark Bulk Load: Need to write HFiles to tmp location then rename to protect from Spark Executor Failures
[ https://issues.apache.org/jira/browse/HBASE-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15271: Resolution: Fixed Release Note: When using the bulk load helper provided by the hbase-spark module, output files will now be written into temporary files and only made available when the executor has successfully completed. Previously, failed executors would leave their files in place in a way that would be picked up by a bulk load command. This caused retried failures to include spurious copies of some cells. Status: Resolved (was: Patch Available) Pushed. Thanks Ted M for the fix. Please close your reviewboard as "submitted". > Spark Bulk Load: Need to write HFiles to tmp location then rename to protect > from Spark Executor Failures > - > > Key: HBASE-15271 > URL: https://issues.apache.org/jira/browse/HBASE-15271 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 2.0.0 > > Attachments: HBASE-15271.1.patch, HBASE-15271.2.patch, > HBASE-15271.3.patch, HBASE-15271.4.patch > > > With the current code if an executor failure before the HFile is close it > will cause problems. This jira will have the files first write out to a file > that starts with an underscore. Then when the HFile is complete it will be > renamed and the underscore will be removed. > The underscore is important because the load bulk functionality will skip > files with an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13372) Unit tests for SplitTransaction and RegionMergeTransaction listeners
[ https://issues.apache.org/jira/browse/HBASE-13372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186294#comment-15186294 ] Gabor Liptak commented on HBASE-13372: -- Any other changes you would like to see for this? Thanks > Unit tests for SplitTransaction and RegionMergeTransaction listeners > > > Key: HBASE-13372 > URL: https://issues.apache.org/jira/browse/HBASE-13372 > Project: HBase > Issue Type: Test >Affects Versions: 2.0.0, 1.1.0 >Reporter: Andrew Purtell >Assignee: Gabor Liptak > Labels: beginner > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13372.2.patch, HBASE-13372.3.patch, > HBASE-13372.4.patch > > > We have new Listener interfaces in SplitTransaction and > RegionMergeTransaction. There are no use cases for these yet, nor unit tests. > We should have unit tests for these that do something just a bit nontrivial > so as to provide a useful example. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186288#comment-15186288 ] Sean Busbey commented on HBASE-6721: thanks for finding an edge case for yetus. ;) I'll spin up a clean build environment and see what yetus does there. > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch, > hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch, > hbase-6721-v25.patch, hbase-6721-v26.patch, hbase-6721-v26_draft1.patch, > hbase-6721-v27.patch, hbase-6721-v27.patch, hbase-6721-v27.patch.txt, > hbase-6721-v28.patch, hbase-6721-v28.patch, hbase-6721-v29.patch, > immediateAssignments Sequence Diagram.svg, randomAssignment Sequence > Diagram.svg, retainAssignment Sequence Diagram.svg, roundRobinAssignment > Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186272#comment-15186272 ] Francis Liu commented on HBASE-6721: [~busbey] looks like the build timed out again but this time because each module was run twice? > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, hbase-6721-v17.patch, > hbase-6721-v18.patch, hbase-6721-v19.patch, hbase-6721-v20.patch, > hbase-6721-v21.patch, hbase-6721-v22.patch, hbase-6721-v23.patch, > hbase-6721-v25.patch, hbase-6721-v26.patch, hbase-6721-v26_draft1.patch, > hbase-6721-v27.patch, hbase-6721-v27.patch, hbase-6721-v27.patch.txt, > hbase-6721-v28.patch, hbase-6721-v28.patch, hbase-6721-v29.patch, > immediateAssignments Sequence Diagram.svg, randomAssignment Sequence > Diagram.svg, retainAssignment Sequence Diagram.svg, roundRobinAssignment > Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14845) hbase-server leaks jdk.tools dependency to mapreduce consumers
[ https://issues.apache.org/jira/browse/HBASE-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14845: Status: In Progress (was: Patch Available) good catch on TestInterfaceAudience* > hbase-server leaks jdk.tools dependency to mapreduce consumers > -- > > Key: HBASE-14845 > URL: https://issues.apache.org/jira/browse/HBASE-14845 > Project: HBase > Issue Type: Bug > Components: build, dependencies >Affects Versions: 1.0.3, 1.1.2, 1.2.0, 0.98.14, 2.0.0, 1.3.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > Attachments: HBASE-14845.1.patch > > > HBASE-13963 / HBASE-14844 take care of removing leaks of our dependency on > jdk-tools. > Until we move the mapreduce support classes out of hbase-server > (HBASE-11843), we need to also avoid leaking the dependency from that module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15430) Failed taking snapshot
[
https://issues.apache.org/jira/browse/HBASE-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JunHo Cho updated HBASE-15430:
--
Attachment: hbase-15430.patch
> Failed taking snapshot
> ---
>
> Key: HBASE-15430
> URL: https://issues.apache.org/jira/browse/HBASE-15430
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 0.98.11
>Reporter: JunHo Cho
>Priority: Critical
> Attachments: hbase-15430.patch
>
>
> the size of a protobuf message is 64MB (default). but the size of snapshot
> meta is over 64MB.
> Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed
> taking snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to
> exception:Protocol message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size
> limit.:com.google.protobuf.InvalidProtocolBufferException: Protocol message
> was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
> increase the size limit.
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
> ... 10 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
> at
> com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
> at
> org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
> at
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
> at
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> at
> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186240#comment-15186240
]
stack commented on HBASE-15392:
---
I expected TestStoreScanner to fail until I figure what expected behavior is.
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch,
> HBASE-15392_suggest.patch, no_optimize.patch, no_optimize.patch, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:279)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:806)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:795)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:624)
> at
> org.apac
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186238#comment-15186238
]
Hadoop QA commented on HBASE-15392:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 28s
{color} | {color:red} hbase-server: patch generated 7 new + 178 unchanged - 5
fixed = 185 total (was 183) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 33s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 32s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 207m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.hbase.ipc.TestSimpleRpcScheduler |
| JDK v1.7.0_95 Failed junit tests | hadoop.hbase.io.hfile.TestCacheConfig |
| | hadoop.hbase.regionserver.TestStoreScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-03-08 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12792081/15392v4.patch |
| JIRA Issue | HBA
[jira] [Created] (HBASE-15430) Failed taking snapshot
JunHo Cho created HBASE-15430:
-
Summary: Failed taking snapshot
Key: HBASE-15430
URL: https://issues.apache.org/jira/browse/HBASE-15430
Project: HBase
Issue Type: Bug
Components: snapshots
Affects Versions: 0.98.11
Reporter: JunHo Cho
Priority: Critical
the size of a protobuf message is 64MB (default). but the size of snapshot meta
is over 64MB.
Caused by: com.google.protobuf.InvalidProtocolBufferException via Failed taking
snapshot { ss=snapshot_xxx table=xxx type=FLUSH } due to exception:Protocol
message was too large. May be malicious. Use CodedInputStream.setSizeLimit()
to increase the size limit.:com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large. May be malicious. Use
CodedInputStream.setSizeLimit() to increase the size limit.
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:83)
at
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:307)
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:341)
... 10 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message
was too large. May be malicious. Use CodedInputStream.setSizeLimit() to
increase the size limit.
at
com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
at
com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
at
com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:811)
at
com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:329)
at
org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3767)
at
org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo.(HBaseProtos.java:3699)
at
org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3815)
at
org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$RegionInfo$1.parsePartialFrom(HBaseProtos.java:3810)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1152)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest.(SnapshotProtos.java:1094)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3858)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.(SnapshotProtos.java:3792)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at
org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:106
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15364) Fix unescaped < characters in Javadoc
[
https://issues.apache.org/jira/browse/HBASE-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-15364:
Resolution: Fixed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, [~gliptak]
> Fix unescaped < characters in Javadoc
> -
>
> Key: HBASE-15364
> URL: https://issues.apache.org/jira/browse/HBASE-15364
> Project: HBase
> Issue Type: Bug
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Gabor Liptak
> Labels: newbie
> Fix For: 2.0.0
>
> Attachments: HBASE-15364.1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/28/artifact/link_report/index.html:
>
> {code}
> host: hbase.apache.org
> date: Mon, 29-Feb-2016 12:06:21 (local)
> Linklint version: 2.3.5_ingo_020
> #
> # warn2 warnings (cross referenced)
> #
> unquoted "<" in <0.90.5, <0.90.5, <
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> unquoted "<" in <0.92.0) a master
> res
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15364) Fix unescaped < characters in Javadoc
[
https://issues.apache.org/jira/browse/HBASE-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186199#comment-15186199
]
Misty Stanley-Jones commented on HBASE-15364:
-
Looks great, thanks!
> Fix unescaped < characters in Javadoc
> -
>
> Key: HBASE-15364
> URL: https://issues.apache.org/jira/browse/HBASE-15364
> Project: HBase
> Issue Type: Bug
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Gabor Liptak
> Labels: newbie
> Attachments: HBASE-15364.1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/28/artifact/link_report/index.html:
>
> {code}
> host: hbase.apache.org
> date: Mon, 29-Feb-2016 12:06:21 (local)
> Linklint version: 2.3.5_ingo_020
> #
> # warn2 warnings (cross referenced)
> #
> unquoted "<" in <0.90.5, <0.90.5, <
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> unquoted "<" in <0.92.0) a master
> res
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15364) Fix unescaped < characters in Javadoc
[
https://issues.apache.org/jira/browse/HBASE-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-15364:
Hadoop Flags: Reviewed
> Fix unescaped < characters in Javadoc
> -
>
> Key: HBASE-15364
> URL: https://issues.apache.org/jira/browse/HBASE-15364
> Project: HBase
> Issue Type: Bug
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Gabor Liptak
> Labels: newbie
> Attachments: HBASE-15364.1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/28/artifact/link_report/index.html:
>
> {code}
> host: hbase.apache.org
> date: Mon, 29-Feb-2016 12:06:21 (local)
> Linklint version: 2.3.5_ingo_020
> #
> # warn2 warnings (cross referenced)
> #
> unquoted "<" in <0.90.5, <0.90.5, <
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> unquoted "<" in <0.92.0) a master
> res
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15364) Fix unescaped < characters in Javadoc
[
https://issues.apache.org/jira/browse/HBASE-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Liptak updated HBASE-15364:
-
Assignee: Gabor Liptak
Release Note: HBASE-15364 Fix unescaped < and > characters in Javadoc
Status: Patch Available (was: Open)
> Fix unescaped < characters in Javadoc
> -
>
> Key: HBASE-15364
> URL: https://issues.apache.org/jira/browse/HBASE-15364
> Project: HBase
> Issue Type: Bug
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Gabor Liptak
> Labels: newbie
> Attachments: HBASE-15364.1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/28/artifact/link_report/index.html:
>
> {code}
> host: hbase.apache.org
> date: Mon, 29-Feb-2016 12:06:21 (local)
> Linklint version: 2.3.5_ingo_020
> #
> # warn2 warnings (cross referenced)
> #
> unquoted "<" in <0.90.5, <0.90.5, <
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> unquoted "<" in <0.92.0) a master
> res
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15364) Fix unescaped < characters in Javadoc
[
https://issues.apache.org/jira/browse/HBASE-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Liptak updated HBASE-15364:
-
Attachment: HBASE-15364.1.patch
> Fix unescaped < characters in Javadoc
> -
>
> Key: HBASE-15364
> URL: https://issues.apache.org/jira/browse/HBASE-15364
> Project: HBase
> Issue Type: Bug
> Components: API, documentation
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
> Labels: newbie
> Attachments: HBASE-15364.1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/28/artifact/link_report/index.html:
>
> {code}
> host: hbase.apache.org
> date: Mon, 29-Feb-2016 12:06:21 (local)
> Linklint version: 2.3.5_ingo_020
> #
> # warn2 warnings (cross referenced)
> #
> unquoted "<" in <0.90.5, <0.90.5, <
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> unquoted "<" in <0.92.0) a master
> res
> occurred in
> /devapidocs/org/apache/hadoop/hbase/util/HBaseFsck.html
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14845) hbase-server leaks jdk.tools dependency to mapreduce consumers
[
https://issues.apache.org/jira/browse/HBASE-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186187#comment-15186187
]
Hadoop QA commented on HBASE-14845:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 58s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 35s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 8s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 19s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 16s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 57s {color}
| {color:red} hbase-client in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 57s {color}
| {color:red} hbase-client in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 8s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 14s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 17s {color}
| {color:red} root in the patch failed with JDK v1.8.0_74. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 14s {color}
| {color:red} hbase-client in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 14s {color}
| {color:red} hbase-client in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 53s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color}
[jira] [Commented] (HBASE-15271) Spark Bulk Load: Need to write HFiles to tmp location then rename to protect from Spark Executor Failures
[ https://issues.apache.org/jira/browse/HBASE-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186180#comment-15186180 ] Ted Malaska commented on HBASE-15271: - Hey [~busbey] the current tests will test that everything still works the same with the rename added. It don't however test that a rename happened. > Spark Bulk Load: Need to write HFiles to tmp location then rename to protect > from Spark Executor Failures > - > > Key: HBASE-15271 > URL: https://issues.apache.org/jira/browse/HBASE-15271 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 2.0.0 > > Attachments: HBASE-15271.1.patch, HBASE-15271.2.patch, > HBASE-15271.3.patch, HBASE-15271.4.patch > > > With the current code if an executor failure before the HFile is close it > will cause problems. This jira will have the files first write out to a file > that starts with an underscore. Then when the HFile is complete it will be > renamed and the underscore will be removed. > The underscore is important because the load bulk functionality will skip > files with an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15429) Add a split policy for busy regions
Ashu Pachauri created HBASE-15429: - Summary: Add a split policy for busy regions Key: HBASE-15429 URL: https://issues.apache.org/jira/browse/HBASE-15429 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Ashu Pachauri Assignee: Ashu Pachauri Fix For: 2.0.0, 1.3.0 Currently, all region split policies are based on size of the region. However, in certain cases, it is a wise choice to make a split decision based on number of requests to the region and split busy regions. A crude metric is that if a region blocks writes often and throws RegionTooBusyExceoption, it's probably a good idea to split it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186178#comment-15186178 ] Andrew Purtell commented on HBASE-15130: [~davelatham] it's my understanding that [~churromorales] has a new backport patch that avoids touching interfaces used by Phoenix yet provides the same client facing API. We will endeavor to get it in to 0.98.19. > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.19 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15429) Add a split policy for busy regions
[ https://issues.apache.org/jira/browse/HBASE-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-15429: -- Description: Currently, all region split policies are based on size. However, in certain cases, it is a wise choice to make a split decision based on number of requests to the region and split busy regions. A crude metric is that if a region blocks writes often and throws RegionTooBusyExceoption, it's probably a good idea to split it. was: Currently, all region split policies are based on size of the region. However, in certain cases, it is a wise choice to make a split decision based on number of requests to the region and split busy regions. A crude metric is that if a region blocks writes often and throws RegionTooBusyExceoption, it's probably a good idea to split it. > Add a split policy for busy regions > --- > > Key: HBASE-15429 > URL: https://issues.apache.org/jira/browse/HBASE-15429 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0 > > > Currently, all region split policies are based on size. However, in certain > cases, it is a wise choice to make a split decision based on number of > requests to the region and split busy regions. > A crude metric is that if a region blocks writes often and throws > RegionTooBusyExceoption, it's probably a good idea to split it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186176#comment-15186176 ] Andrew Purtell commented on HBASE-15130: Reverted from 0.98.18 for now. > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.19 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15271) Spark Bulk Load: Need to write HFiles to tmp location then rename to protect from Spark Executor Failures
[
https://issues.apache.org/jira/browse/HBASE-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186170#comment-15186170
]
Hadoop QA commented on HBASE-15271:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} master passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
38m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 27s
{color} | {color:green} hbase-spark in the patch passed with JDK v1.8.0_74.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hbase-spark in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 10s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-03-08 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12788735/HBASE-15271.4.patch |
| JIRA Issue | HBASE-15271 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux b9dfc27cfefd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 46cc3d4 |
| JDK v1.7.0_95 Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/901/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/901/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Spark Bulk Load: Need to write HFiles to tmp location then rename to protect
> from Spark Executor Failures
> -
>
> Key: HBASE-15271
> URL: https://issues.apache.org/jira/browse/HBASE-15271
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ted Malaska
>Assignee: Ted Malaska
> Fix For: 2.0.0
>
> Attachments: HBASE-15271.1.patch, HBASE-15271.2.patch,
> HBASE-15271.3.patch, HBASE-15271.4.patch
>
>
> With the current code if an executor failure before the HFile is close it
> will cause problems. This jira w
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186155#comment-15186155 ] Dave Latham commented on HBASE-15130: - We're using this functionality but would not be quickly affected by a revert, as it is in our local build. Next time we update we hope this is available upstream so it would be great to be back in for a later release, but understand the current impact on phoenix. > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.19 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15130: --- Fix Version/s: (was: 0.98.18) 0.98.19 > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.19 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15181) A simple implementation of date based tiered compaction
[ https://issues.apache.org/jira/browse/HBASE-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15181: --- Fix Version/s: (was: 0.98.19) 0.98.18 > A simple implementation of date based tiered compaction > --- > > Key: HBASE-15181 > URL: https://issues.apache.org/jira/browse/HBASE-15181 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Clara Xiong >Assignee: Clara Xiong > Fix For: 2.0.0, 1.3.0, 0.98.18 > > Attachments: HBASE-15181-0.98-ADD.patch, HBASE-15181-0.98.patch, > HBASE-15181-0.98.v4.patch, HBASE-15181-98.patch, HBASE-15181-ADD.patch, > HBASE-15181-branch-1.patch, HBASE-15181-master-v1.patch, > HBASE-15181-master-v2.patch, HBASE-15181-master-v3.patch, > HBASE-15181-master-v4.patch, HBASE-15181-v1.patch, HBASE-15181-v2.patch > > > This is a simple implementation of date-based tiered compaction similar to > Cassandra's for the following benefits: > 1. Improve date-range-based scan by structuring store files in date-based > tiered layout. > 2. Reduce compaction overhead. > 3. Improve TTL efficiency. > Perfect fit for the use cases that: > 1. has mostly date-based date write and scan and a focus on the most recent > data. > 2. never or rarely deletes data. > Out-of-order writes are handled gracefully. Time range overlapping among > store files is tolerated and the performance impact is minimized. > Configuration can be set at hbase-site.xml or overriden at per-table or > per-column-famly level by hbase shell. > Design spec is at > https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLKIqL6T-oHoRLZ323MG_uy8/edit?usp=sharing > Results in our production is at > https://docs.google.com/document/d/1GqRtQZMMkTEWOijZc8UCTqhACNmdxBSjtAQSYIWsmGU/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15417) Calls to ObserverContext#bypass in a region observer's prePut method are inconsistent
[ https://issues.apache.org/jira/browse/HBASE-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186067#comment-15186067 ] Harry Harpham commented on HBASE-15417: --- Thank you for the response! > Calls to ObserverContext#bypass in a region observer's prePut method are > inconsistent > - > > Key: HBASE-15417 > URL: https://issues.apache.org/jira/browse/HBASE-15417 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Reporter: Harry Harpham >Priority: Minor > > Calling ctx.bypass(), where ctx is the ObserverContext object passed in to > the region observer's prePut method, results in some inconsistent behavior. > If every other put in the batch is also bypassed, the region observer sees > none of these in its postPut method. If there is at least one other put > which is not bypassed, the region observer sees all of the puts in the batch > _including those which were bypassed_. > The end result is that, after bypassing a put, that put may or may not end up > in the region observer's postPut method. This behavior is dependent solely > on which other puts the bypassed put is batched together with. > I tried to find existing tickets for this issue, but was unable to. > Apologies if I missed something. The closest issues I could find were > HBASE-4331 and HBASE-11503, but those didn't seem to quite hit it. > Additionally, I threw together a quick demonstration of this issue: > https://github.com/hwh33/bypass-inconsistency-demo. You can run that demo in > memory using the testing utility or against a running cluster. I actually > haven't had time to test it against a cluster though, so you may encounter > bugs if running in that mode (but hopefully not!). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15271) Spark Bulk Load: Need to write HFiles to tmp location then rename to protect from Spark Executor Failures
[ https://issues.apache.org/jira/browse/HBASE-15271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186061#comment-15186061 ] Sean Busbey commented on HBASE-15271: - [~ted.m] does some pre-existing test cover this functionality? > Spark Bulk Load: Need to write HFiles to tmp location then rename to protect > from Spark Executor Failures > - > > Key: HBASE-15271 > URL: https://issues.apache.org/jira/browse/HBASE-15271 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 2.0.0 > > Attachments: HBASE-15271.1.patch, HBASE-15271.2.patch, > HBASE-15271.3.patch, HBASE-15271.4.patch > > > With the current code if an executor failure before the HFile is close it > will cause problems. This jira will have the files first write out to a file > that starts with an underscore. Then when the HFile is complete it will be > renamed and the underscore will be removed. > The underscore is important because the load bulk functionality will skip > files with an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186054#comment-15186054
]
Ted Yu commented on HBASE-15406:
SwitchStateTracker (running in master JVM) creates the (ephemeral) znode.
We need to find a way to decouple the creation of (ephemeral) znode from
SwitchStateTracker. Otherwise after hbck exits, the znode still lives.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.v1.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
