[jira] [Commented] (HBASE-11219) HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop
[ https://issues.apache.org/jira/browse/HBASE-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004400#comment-14004400 ] Mikhail Antonov commented on HBASE-11219: - *patch looks good to me HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop --- Key: HBASE-11219 URL: https://issues.apache.org/jira/browse/HBASE-11219 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Attachments: 11219-v1.txt As Andrew showed in the attachment to HBASE-11165, ClusterStatusProtos$RegionLoad$Builder took 25MB heap space. buildServerLoad() calls createRegionLoad() for each region on the region server. One instance of ClusterStatusProtos$RegionLoad$Builder can be created outside the loop and passed to createRegionLoad(). Thanks Andrew for the finding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11202) Cleanup on HRegion class
[ https://issues.apache.org/jira/browse/HBASE-11202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004427#comment-14004427 ] Anoop Sam John commented on HBASE-11202: bq.This was added by Jeffrey Zhong it seems. Jeffrey, should we keep them? No not by Jeffrey. This was added by the commit for the issue HBASE-9645. [~stack] seems u had done that commit. Remembering why these 2 APIs added boss? Cleanup on HRegion class Key: HBASE-11202 URL: https://issues.apache.org/jira/browse/HBASE-11202 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.99.0 Attachments: 11202.v1.patch This is mostly trivial stuff - remove some methods not used - typos - remove some @param w/o any info - change the code that uses deprecated methods The only non trivial change is when we get the store from a cell: instead of using the map, we iterate on the key set. Likely, it would be better to hava a sorted array instead of a Map, as the number of store is fixed. Could be done in a later patch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10417) index is not incremented in PutSortReducer#reduce()
[
https://issues.apache.org/jira/browse/HBASE-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004452#comment-14004452
]
Lars Hofhansl commented on HBASE-10417:
---
Not at a computer right now... Is this an issue for 0.94 as well?
index is not incremented in PutSortReducer#reduce()
---
Key: HBASE-10417
URL: https://issues.apache.org/jira/browse/HBASE-10417
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Minor
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-10417.patch
Starting at line 76:
{code}
int index = 0;
for (KeyValue kv : map) {
context.write(row, kv);
if (index 0 index % 100 == 0)
context.setStatus(Wrote + index);
{code}
index is a variable inside while loop that is never incremented.
The condition index 0 cannot be true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-7623) Username is not available for HConnectionManager to use in HConnectionKey
[
https://issues.apache.org/jira/browse/HBASE-7623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yogesh bedekar updated HBASE-7623:
--
Attachment: yogesh_bedekar.vcf
I am using version 3.4.5.
Username is not available for HConnectionManager to use in HConnectionKey
-
Key: HBASE-7623
URL: https://issues.apache.org/jira/browse/HBASE-7623
Project: HBase
Issue Type: Improvement
Components: Client, security
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: pom.xml, trunk-7623.patch, yogesh_bedekar.vcf,
yogesh_bedekar.vcf, yogesh_bedekar.vcf, yogesh_bedekar.vcf,
yogesh_bedekar.vcf, yogesh_bedekar.vcf
Sometimes, some non-IOException prevents User.getCurrent() to get a username.
It makes it impossible to create a HConnection. We should catch all
exception here:
{noformat}
try {
User currentUser = User.getCurrent();
if (currentUser != null) {
username = currentUser.getName();
}
} catch (IOException ioe) {
LOG.warn(Error obtaining current user, skipping username in
HConnectionKey,
ioe);
}
{noformat}
Not just IOException, so that client can move forward.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11220) Add listeners to ServerManager and AssignmentManager
Matteo Bertozzi created HBASE-11220: --- Summary: Add listeners to ServerManager and AssignmentManager Key: HBASE-11220 URL: https://issues.apache.org/jira/browse/HBASE-11220 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Add support for listeners to ServerManager and AssignmentManager. This will allows to get notified about servers added/removed or regions added/removed/moved. I'm planning to use this in the MasterProcedureManager. Since we are starting using the Procedures for distributed operations, we must add support for RS joining or Regions moving. At the moment the operation on the moving set of RSs is lost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11220) Add listeners to ServerManager and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-11220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-11220: Issue Type: Improvement (was: Bug) Add listeners to ServerManager and AssignmentManager Key: HBASE-11220 URL: https://issues.apache.org/jira/browse/HBASE-11220 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Add support for listeners to ServerManager and AssignmentManager. This will allows to get notified about servers added/removed or regions added/removed/moved. I'm planning to use this in the MasterProcedureManager. Since we are starting using the Procedures for distributed operations, we must add support for RS joining or Regions moving. At the moment the operation on the moving set of RSs is lost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11221) Make the MasterProcedure aware of RS and Regions changes
Matteo Bertozzi created HBASE-11221: --- Summary: Make the MasterProcedure aware of RS and Regions changes Key: HBASE-11221 URL: https://issues.apache.org/jira/browse/HBASE-11221 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Use the ServerManager and AssignmentManager notifications HBASE-11220 to detect new server or regions movement and make the in-progress Procedure aware of those changes. At the moment the operation on the moving set of RSs is lost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[
https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004563#comment-14004563
]
Nick Dimiduk commented on HBASE-11216:
--
Nit: parameter name should be camel case {{valueLength}}
{noformat}
+ private static String calculateMbps(int rows, int value_length, long timeMs)
{
{noformat}
Otherwise, +1. I think [~nkeywal] is looking into the jenkins/java issues.
PerformanceEvaluation should provide an option to modify the value length.
--
Key: HBASE-11216
URL: https://issues.apache.org/jira/browse/HBASE-11216
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Jean-Marc Spaggiari
Assignee: Jean-Marc Spaggiari
Priority: Minor
Attachments: HBASE-11216-v0-trunk.patch
All in the title.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11215) Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)
[ https://issues.apache.org/jira/browse/HBASE-11215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004567#comment-14004567 ] Nick Dimiduk commented on HBASE-11215: -- +1 Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail) - Key: HBASE-11215 URL: https://issues.apache.org/jira/browse/HBASE-11215 Project: HBase Issue Type: Bug Reporter: Carter Assignee: Carter Attachments: HBASE_11215.patch The JavaDoc for HTableInterface#setAutoFlush(boolean autoFlush, boolean clearBufferOnFail) says Setting clearBufferOnFail to false is deprecated since 0.96. If that's the case, this method should be deprecated as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9507) Promote methods of WALActionsListener to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004564#comment-14004564 ] Nick Dimiduk commented on HBASE-9507: - LogRoller and Replication both implement this interface today. I don't see it used directly in Phoenix or HBase Indexer. Promote methods of WALActionsListener to WALObserver Key: HBASE-9507 URL: https://issues.apache.org/jira/browse/HBASE-9507 Project: HBase Issue Type: Brainstorming Components: Coprocessors, wal Reporter: Nick Dimiduk Priority: Minor Fix For: 0.99.0 The interface exposed by WALObserver is quite minimal. To implement anything of significance based on WAL events, WALActionsListener (at a minimum) is required. This is demonstrated by the implementation of the replication feature (not currently possible with coprocessors) and the corresponding interface exploitation that is the [Side-Effect Processor|https://github.com/NGDATA/hbase-indexer/tree/master/hbase-sep]. Consider promoting the interface of WALActionsListener into WALObserver. This goes a long way to being able refactor replication into a coprocessor. This also removes the duplicate code path for listeners because they're already available via coprocessor hook. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10818) Add integration test for bulkload with replicas
[
https://issues.apache.org/jira/browse/HBASE-10818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004573#comment-14004573
]
Nick Dimiduk commented on HBASE-10818:
--
{quote}
bq. You should set the sleepTime in SlowMeCopro to something like 10 seconds.
It defaults to 0.
This is tricky. There isn't a way to modify the state of that coproc on a real
cluster. Maybe an endpoint could be added?
{quote}
No solution to this question as of yet?
Add integration test for bulkload with replicas
---
Key: HBASE-10818
URL: https://issues.apache.org/jira/browse/HBASE-10818
Project: HBase
Issue Type: Sub-task
Affects Versions: hbase-10070
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: hbase-10070
Attachments: 10818-7.txt, HBASE-10818.00.patch, HBASE-10818.01.patch,
HBASE-10818.02.patch, IntegrationTestBulkLoad_replicas.log
Should verify bulkload is not affected by region replicas.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11157) [hbck] NotServingRegionException: Received close for regionName but we are not serving it
[ https://issues.apache.org/jira/browse/HBASE-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004580#comment-14004580 ] Kashif J S commented on HBASE-11157: IMO, For the patch you can explicitly catch (NotServingRegionException e) instead of IOException [hbck] NotServingRegionException: Received close for regionName but we are not serving it --- Key: HBASE-11157 URL: https://issues.apache.org/jira/browse/HBASE-11157 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.13 Reporter: dailidong Priority: Trivial Attachments: HBASE-11157.patch if hbck close a region then meet a NotServerRegionException,hbck will hang up . we will close the region on the regionserver, but this regionserver is not serving the region, so we should try catch this exception. Trying to fix unassigned region... Exception in thread main org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: Received close for regionName but we are not serving it at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3204) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3185) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1012) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:87) at com.sun.proxy.$Proxy7.closeRegion(Unknown Source) at org.apache.hadoop.hbase.util.HBaseFsckRepair.closeRegionSilentlyAndWait(HBaseFsckRepair.java:150) at org.apache.hadoop.hbase.util.HBaseFsck.closeRegion(HBaseFsck.java:1565) at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1704) at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1406) at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:419) at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:438) at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3670) at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3489) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3483) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11222) Add integration test for visibility deletes
ramkrishna.s.vasudevan created HBASE-11222: -- Summary: Add integration test for visibility deletes Key: HBASE-11222 URL: https://issues.apache.org/jira/browse/HBASE-11222 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.3 Once HBASE-10885 gets checked in, create integration tests to verify the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[ https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-11216: Status: Open (was: Patch Available) PerformanceEvaluation should provide an option to modify the value length. -- Key: HBASE-11216 URL: https://issues.apache.org/jira/browse/HBASE-11216 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-11216-v0-trunk.patch, HBASE-11216-v1-trunk.patch All in the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[ https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-11216: Status: Patch Available (was: Open) Damn it! I missed this one. Thanks for look at it. Just attached an updated version. Let's see if Jenkins is going better... PerformanceEvaluation should provide an option to modify the value length. -- Key: HBASE-11216 URL: https://issues.apache.org/jira/browse/HBASE-11216 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-11216-v0-trunk.patch, HBASE-11216-v1-trunk.patch All in the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[ https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-11216: Attachment: HBASE-11216-v1-trunk.patch PerformanceEvaluation should provide an option to modify the value length. -- Key: HBASE-11216 URL: https://issues.apache.org/jira/browse/HBASE-11216 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-11216-v0-trunk.patch, HBASE-11216-v1-trunk.patch All in the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11223) Limit the actions number of a call in the batch
Liu Shaohui created HBASE-11223:
---
Summary: Limit the actions number of a call in the batch
Key: HBASE-11223
URL: https://issues.apache.org/jira/browse/HBASE-11223
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Huge batch operation will make regionserver crash for GC.
The extreme code like this:
{code}
final ListDelete deletes = new ArrayListDelete();
final long rows = 400;
for (long i = 0; i rows; ++i) {
deletes.add(new Delete(Bytes.toBytes(i)));
}
table.delete(deletes);
{code}
We should limit the actions number of a call in the batch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004641#comment-14004641 ] Nick Dimiduk commented on HBASE-9857: - This is looking very nice. I think we'll want to add a similar prefetch feature for index and bloom blocks, but that can come in a later revision. +1 Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11223) Limit the actions number of a call in the batch
[
https://issues.apache.org/jira/browse/HBASE-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004651#comment-14004651
]
Nicolas Liochon commented on HBASE-11223:
-
This should be managed in the server rather than the client: there are other
clients around. There is already some code in the server around this category
of issue (gc due to request queue, this kind of things).
Limit the actions number of a call in the batch
Key: HBASE-11223
URL: https://issues.apache.org/jira/browse/HBASE-11223
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Huge batch operation will make regionserver crash for GC.
The extreme code like this:
{code}
final ListDelete deletes = new ArrayListDelete();
final long rows = 400;
for (long i = 0; i rows; ++i) {
deletes.add(new Delete(Bytes.toBytes(i)));
}
table.delete(deletes);
{code}
We should limit the actions number of a call in the batch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11206) Enable automagically tweaking memstore and blockcache sizes
[ https://issues.apache.org/jira/browse/HBASE-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004656#comment-14004656 ] Nick Dimiduk commented on HBASE-11206: -- Very well. We'll deal with the mixed onheap + offheap stuff when we get to it. Enable automagically tweaking memstore and blockcache sizes --- Key: HBASE-11206 URL: https://issues.apache.org/jira/browse/HBASE-11206 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Fix For: 0.99.0 HBASE-5349 Automagically tweak global memstore and block cache sizes based on workload adds a nice new feature. It is off by default. Lets turn it on for 0.99. Liang Xie is concerned that automatically shifting blockcache and memstore sizes could wreck havoc w/ GC'ing in low-latency serving situation -- a valid concern -- but lets enable this feature in 0.99 and see how it does. Can always disable it before 1.0 if a problem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[ https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-11216: Status: Open (was: Patch Available) If size nor perClientRunRows are not specified need to calculate size based on valueLength... I will update the patch. PerformanceEvaluation should provide an option to modify the value length. -- Key: HBASE-11216 URL: https://issues.apache.org/jira/browse/HBASE-11216 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-11216-v0-trunk.patch, HBASE-11216-v1-trunk.patch All in the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10933) hbck -fixHdfsOrphans is not working properly it throws null pointer exception
[
https://issues.apache.org/jira/browse/HBASE-10933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kashif J S updated HBASE-10933:
---
Attachment: HBASE-10933-0.94-v2.patch
Modified Junit TCs for 0.94 version for TestOfflineMetaRebuildHole
TestOfflineMetaRebuildOverlap. Please review the v2 patch for 0.94.
hbck -fixHdfsOrphans is not working properly it throws null pointer exception
-
Key: HBASE-10933
URL: https://issues.apache.org/jira/browse/HBASE-10933
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.16, 0.98.2
Reporter: Deepak Sharma
Assignee: Kashif J S
Priority: Critical
Fix For: 0.99.0, 0.94.21
Attachments: HBASE-10933-0.94-v1.patch, HBASE-10933-0.94-v2.patch,
HBASE-10933-trunk-v1.patch, TestResults-0.94.txt, TestResults-trunk.txt
if we regioninfo file is not existing in hbase region then if we run hbck
repair or hbck -fixHdfsOrphans
then it is not able to resolve this problem it throws null pointer exception
{code}
2014-04-08 20:11:49,750 INFO [main] util.HBaseFsck
(HBaseFsck.java:adoptHdfsOrphans(470)) - Attempting to handle orphan hdfs
dir:
hdfs://10.18.40.28:54310/hbase/TestHdfsOrphans1/5a3de9ca65e587cb05c9384a3981c950
java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$000(HBaseFsck.java:1939)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:497)
at
org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:471)
at
org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:591)
at
org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:369)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:447)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3769)
at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3587)
at
com.huawei.isap.test.smartump.hadoop.hbase.HbaseHbckRepair.repairToFixHdfsOrphans(HbaseHbckRepair.java:244)
at
com.huawei.isap.test.smartump.hadoop.hbase.HbaseHbckRepair.setUp(HbaseHbckRepair.java:84)
at junit.framework.TestCase.runBare(TestCase.java:132)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
{code}
problem i got it is because since in HbaseFsck class
{code}
private void adoptHdfsOrphan(HbckInfo hi)
{code}
we are intializing tableinfo using SortedMapString, TableInfo tablesInfo
object
{code}
TableInfo tableInfo = tablesInfo.get(tableName);
{code}
but in private SortedMapString, TableInfo loadHdfsRegionInfos()
{code}
for (HbckInfo hbi: hbckInfos) {
if (hbi.getHdfsHRI() == null) {
// was an orphan
continue;
}
{code}
we have check if a region is orphan then that table will can not be added in
SortedMapString, TableInfo tablesInfo
so later while using this we get null pointer exception
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11211) LoadTestTool option for specifying number of regions per server
[
https://issues.apache.org/jira/browse/HBASE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004659#comment-14004659
]
Nick Dimiduk commented on HBASE-11211:
--
Looks like a println snuck in.
{noformat}
+System.out.println(Regions per server: + regionsPerServer);
{noformat}
Otherwise, +1.
LoadTestTool option for specifying number of regions per server
---
Key: HBASE-11211
URL: https://issues.apache.org/jira/browse/HBASE-11211
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Trivial
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11211-0.98.patch, HBASE-11211-trunk.patch
Add a new LoadTestTool option for specifying number of regions per server.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[ https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-11216: Attachment: HBASE-11216-v2-trunk.patch PerformanceEvaluation should provide an option to modify the value length. -- Key: HBASE-11216 URL: https://issues.apache.org/jira/browse/HBASE-11216 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-11216-v0-trunk.patch, HBASE-11216-v1-trunk.patch, HBASE-11216-v2-trunk.patch All in the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11216) PerformanceEvaluation should provide an option to modify the value length.
[ https://issues.apache.org/jira/browse/HBASE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-11216: Status: Patch Available (was: Open) I have 0.98 patch ready too, but will wait for Jenkins before uploading it. PerformanceEvaluation should provide an option to modify the value length. -- Key: HBASE-11216 URL: https://issues.apache.org/jira/browse/HBASE-11216 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-11216-v0-trunk.patch, HBASE-11216-v1-trunk.patch, HBASE-11216-v2-trunk.patch All in the title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11223) Limit the actions number of a call in the batch
[
https://issues.apache.org/jira/browse/HBASE-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004675#comment-14004675
]
Liu Shaohui commented on HBASE-11223:
-
[~nkeywal]
Good advice!
What about limit the dataLength when reading the request data from the channel
in the rpc server?
Eg, if the dataLength is larger than 2M, just throw an exception.
Limit the actions number of a call in the batch
Key: HBASE-11223
URL: https://issues.apache.org/jira/browse/HBASE-11223
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Huge batch operation will make regionserver crash for GC.
The extreme code like this:
{code}
final ListDelete deletes = new ArrayListDelete();
final long rows = 400;
for (long i = 0; i rows; ++i) {
deletes.add(new Delete(Bytes.toBytes(i)));
}
table.delete(deletes);
{code}
We should limit the actions number of a call in the batch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11224) TestReversibleScanners should verify different DataBlockEncoding's
Ted Yu created HBASE-11224: -- Summary: TestReversibleScanners should verify different DataBlockEncoding's Key: HBASE-11224 URL: https://issues.apache.org/jira/browse/HBASE-11224 Project: HBase Issue Type: Test Reporter: Ted Yu Priority: Minor Currently HFileContextBuilder in the tests doesn't specify DataBlockEncoding. TestReversibleScanners should be parameterized and verify that reversible scanners work with different DataBlockEncoding's -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11223) Limit the actions number of a call in the batch
[
https://issues.apache.org/jira/browse/HBASE-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004682#comment-14004682
]
Nicolas Liochon commented on HBASE-11223:
-
bq. What about limit the dataLength when reading the request data from the
channel in the rpc server?
Yes, I agree.
IIRC, there is some code that does that, so may be this code comes too late and
the rs crashed already, or this code is broken...
Limit the actions number of a call in the batch
Key: HBASE-11223
URL: https://issues.apache.org/jira/browse/HBASE-11223
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Huge batch operation will make regionserver crash for GC.
The extreme code like this:
{code}
final ListDelete deletes = new ArrayListDelete();
final long rows = 400;
for (long i = 0; i rows; ++i) {
deletes.add(new Delete(Bytes.toBytes(i)));
}
table.delete(deletes);
{code}
We should limit the actions number of a call in the batch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11225) Backport fix for HBASE-10417 'index is not incremented in PutSortReducer#reduce()'
Ted Yu created HBASE-11225:
--
Summary: Backport fix for HBASE-10417 'index is not incremented in
PutSortReducer#reduce()'
Key: HBASE-11225
URL: https://issues.apache.org/jira/browse/HBASE-11225
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Fix For: 0.94.20
The problem reported in HBASE-10417 exists in 0.94 code base.
{code}
for (KeyValue kv : map) {
context.write(row, kv);
if (index 0 index % 100 == 0)
context.setStatus(Wrote + index);
}
{code}
This JIRA backports the fix to 0.94.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10417) index is not incremented in PutSortReducer#reduce()
[
https://issues.apache.org/jira/browse/HBASE-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004690#comment-14004690
]
Ted Yu commented on HBASE-10417:
The problem is in 0.94 as well.
Opened HBASE-11225 for backport.
index is not incremented in PutSortReducer#reduce()
---
Key: HBASE-10417
URL: https://issues.apache.org/jira/browse/HBASE-10417
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Minor
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-10417.patch
Starting at line 76:
{code}
int index = 0;
for (KeyValue kv : map) {
context.write(row, kv);
if (index 0 index % 100 == 0)
context.setStatus(Wrote + index);
{code}
index is a variable inside while loop that is never incremented.
The condition index 0 cannot be true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-11225) Backport fix for HBASE-10417 'index is not incremented in PutSortReducer#reduce()'
[
https://issues.apache.org/jira/browse/HBASE-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo Anatoly reassigned HBASE-11225:
---
Assignee: Gustavo Anatoly
Backport fix for HBASE-10417 'index is not incremented in
PutSortReducer#reduce()'
--
Key: HBASE-11225
URL: https://issues.apache.org/jira/browse/HBASE-11225
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Minor
Fix For: 0.94.20
The problem reported in HBASE-10417 exists in 0.94 code base.
{code}
for (KeyValue kv : map) {
context.write(row, kv);
if (index 0 index % 100 == 0)
context.setStatus(Wrote + index);
}
{code}
This JIRA backports the fix to 0.94.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11224) TestReversibleScanners should verify different DataBlockEncoding's
[ https://issues.apache.org/jira/browse/HBASE-11224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11224: -- Priority: Trivial (was: Minor) TestReversibleScanners should verify different DataBlockEncoding's -- Key: HBASE-11224 URL: https://issues.apache.org/jira/browse/HBASE-11224 Project: HBase Issue Type: Test Reporter: Ted Yu Priority: Trivial Currently HFileContextBuilder in the tests doesn't specify DataBlockEncoding. TestReversibleScanners should be parameterized and verify that reversible scanners work with different DataBlockEncoding's -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11217) Race between SplitLogManager task creation + TimeoutMonitor
[
https://issues.apache.org/jira/browse/HBASE-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004748#comment-14004748
]
Jimmy Xiang commented on HBASE-11217:
-
Because log replay is enabled in your test? Make sense.
Race between SplitLogManager task creation + TimeoutMonitor
---
Key: HBASE-11217
URL: https://issues.apache.org/jira/browse/HBASE-11217
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Critical
Fix For: 0.99.0
Attachments: hbase-11217_v1.patch
Some time ago, we reported a test failure in HBASE-11036, which resulted in
already-split and merged regions coming back to life, causing split brain for
region boundaries and resulting in data loss.
It turns out that the root cause was not concurrent online schema change +
region split/merge, but meta log splitting failing and the meta updates
getting lost. This in turn causes the region split/merge information and
assignment to be lost causing large scale data loss.
Logs below shows that the split task for meta log is created, but before the
znode is created, the timeout thread kicks in and sees the unassigned task.
Then it does a get on znode which fails with NoNode (because the znode is not
created yet). This causes the task to be marked complete (setDone(path,
SUCCESS)) which means that the logs are lost. Meta is assigned elsewhere (and
opened with the same seqId as previous) confirming data loss in meta.
{code}
2014-04-16 18:31:26,267 INFO
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2]
handler.MetaServerShutdownHandler: Splitting hbase:meta logs for
hor13n03.gq1.ygridcore.net,60020,1397672668647
2014-04-16 18:31:26,274 DEBUG
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2] master.MasterFileSystem:
Renamed region directory:
hdfs://hor13n01.gq1.ygridcore.net:8020/apps/hbase/data/WALs/hor13n03.gq1.ygridcore.net,60020,1397672668647-splitting
2014-04-16 18:31:26,274 INFO
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2] master.SplitLogManager: dead
splitlog workers [hor13n03.gq1.ygridcore.net,60020,1397672668647]
2014-04-16 18:31:26,276 DEBUG
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2] master.SplitLogManager:
Scheduling batch of logs to split
2014-04-16 18:31:26,276 INFO
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2] master.SplitLogManager:
started splitting 1 logs in
[hdfs://hor13n01.gq1.ygridcore.net:8020/apps/hbase/data/WALs/hor13n03.gq1.ygridcore.net,60020,1397672668647-splitting]
2014-04-16 18:31:26,276 INFO
[hor13n02.gq1.ygridcore.net,6,1397672191204.splitLogManagerTimeoutMonitor]
master.SplitLogManager: total tasks = 1 unassigned = 1
tasks={/hbase/splitWAL/WALs%2Fhor13n03.gq1.ygridcore.net%2C60020%2C1397672668647-splitting%2Fhor13n03.gq1.ygridcore.net%252C60020%252C1397672668647.1397672681632.meta=last_update
= -1 last_version = -
2014-04-16 18:31:26,276 DEBUG
[hor13n02.gq1.ygridcore.net,6,1397672191204.splitLogManagerTimeoutMonitor]
master.SplitLogManager: resubmitting unassigned task(s) after timeout
2014-04-16 18:31:26,277 WARN [main-EventThread]
master.SplitLogManager$GetDataAsyncCallback: task znode
/hbase/splitWAL/WALs%2Fhor13n03.gq1.ygridcore.net%2C60020%2C1397672668647-splitting%2Fhor13n03.gq1.ygridcore.net%252C60020%252C1397672668647.1397672681632.meta
vanished.
2014-04-16 18:31:26,277 INFO [main-EventThread] master.SplitLogManager: Done
splitting
/hbase/splitWAL/WALs%2Fhor13n03.gq1.ygridcore.net%2C60020%2C1397672668647-splitting%2Fhor13n03.gq1.ygridcore.net%252C60020%252C1397672668647.1397672681632.meta
2014-04-16 18:31:26,282 DEBUG [main-EventThread] master.SplitLogManager: put
up splitlog task at znode
/hbase/splitWAL/WALs%2Fhor13n03.gq1.ygridcore.net%2C60020%2C1397672668647-splitting%2Fhor13n03.gq1.ygridcore.net%252C60020%252C1397672668647.1397672681632.meta
2014-04-16 18:31:26,286 WARN
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2] master.SplitLogManager:
returning success without actually splitting and deleting all the log files
in path
hdfs://hor13n01.gq1.ygridcore.net:8020/apps/hbase/data/WALs/hor13n03.gq1.ygridcore.net,60020,1397672668647-splitting
2014-04-16 18:31:26,286 INFO
[MASTER_META_SERVER_OPERATIONS-hor13n02:6-2] master.SplitLogManager:
finished splitting (more than or equal to) 9 bytes in 1 log files in
[hdfs://hor13n01.gq1.ygridcore.net:8020/apps/hbase/data/WALs/hor13n03.gq1.ygridcore.net,60020,1397672668647-splitting]
in 10ms
2014-04-16 18:31:26,290 DEBUG [main-EventThread]
master.SplitLogManager$DeleteAsyncCallback: deleted
[jira] [Updated] (HBASE-10573) Use Netty 4
[ https://issues.apache.org/jira/browse/HBASE-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-10573: Attachment: 10573.add.patch Use Netty 4 --- Key: HBASE-10573 URL: https://issues.apache.org/jira/browse/HBASE-10573 Project: HBase Issue Type: Sub-task Affects Versions: 0.99.0, hbase-10191 Reporter: Andrew Purtell Assignee: Nicolas Liochon Fix For: 0.99.0 Attachments: 10573.add.patch, 10573.patch, 10573.patch, 10573.v3.patch Pull in Netty 4 and sort out the consequences. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10573) Use Netty 4
[ https://issues.apache.org/jira/browse/HBASE-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004755#comment-14004755 ] Nicolas Liochon commented on HBASE-10573: - 10573.add.patch contains what I plan to commit once the migration to git is done. Use Netty 4 --- Key: HBASE-10573 URL: https://issues.apache.org/jira/browse/HBASE-10573 Project: HBase Issue Type: Sub-task Affects Versions: 0.99.0, hbase-10191 Reporter: Andrew Purtell Assignee: Nicolas Liochon Fix For: 0.99.0 Attachments: 10573.add.patch, 10573.patch, 10573.patch, 10573.v3.patch Pull in Netty 4 and sort out the consequences. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004770#comment-14004770 ] Andrew Purtell commented on HBASE-9857: --- The prefetch loads all blocks except meta blocks, so that includes indexes and blooms. Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11211) LoadTestTool option for specifying number of regions per server
[ https://issues.apache.org/jira/browse/HBASE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004775#comment-14004775 ] Andrew Purtell commented on HBASE-11211: bq. Looks like a println snuck in. No that's intentional. LTT prints other variable configuration to stdout when starting. Let me know if you want me to remove this particular instance. LoadTestTool option for specifying number of regions per server --- Key: HBASE-11211 URL: https://issues.apache.org/jira/browse/HBASE-11211 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11211-0.98.patch, HBASE-11211-trunk.patch Add a new LoadTestTool option for specifying number of regions per server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11222) Add integration test for visibility deletes
[ https://issues.apache.org/jira/browse/HBASE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11222. Resolution: Duplicate Fix Version/s: (was: 0.98.3) (was: 0.99.0) Assignee: (was: ramkrishna.s.vasudevan) Dup of HBASE-11039 Add integration test for visibility deletes --- Key: HBASE-11222 URL: https://issues.apache.org/jira/browse/HBASE-11222 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Once HBASE-10885 gets checked in, create integration tests to verify the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004778#comment-14004778 ] Nick Dimiduk commented on HBASE-9857: - Yes of course; I wasn't clear in my comment. I'm suggesting future work would enable only prefetching non-data blocks. Thus guaranteeing indices are always warm. One would want this when you know data size cache size. Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11211) LoadTestTool option for specifying number of regions per server
[ https://issues.apache.org/jira/browse/HBASE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004782#comment-14004782 ] Nick Dimiduk commented on HBASE-11211: -- Very well. No objection. LoadTestTool option for specifying number of regions per server --- Key: HBASE-11211 URL: https://issues.apache.org/jira/browse/HBASE-11211 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11211-0.98.patch, HBASE-11211-trunk.patch Add a new LoadTestTool option for specifying number of regions per server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004787#comment-14004787 ] Andrew Purtell commented on HBASE-9857: --- Commit on hold until the SVN-GIT migration is finished. Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11211) LoadTestTool option for specifying number of regions per server
[ https://issues.apache.org/jira/browse/HBASE-11211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004788#comment-14004788 ] Andrew Purtell commented on HBASE-11211: Commit on hold until the SVN-GIT migration is finished. LoadTestTool option for specifying number of regions per server --- Key: HBASE-11211 URL: https://issues.apache.org/jira/browse/HBASE-11211 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11211-0.98.patch, HBASE-11211-trunk.patch Add a new LoadTestTool option for specifying number of regions per server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-11039: -- Assignee: ramkrishna.s.vasudevan [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 0.98.3 Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9857) Blockcache prefetch option
[ https://issues.apache.org/jira/browse/HBASE-9857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004790#comment-14004790 ] Andrew Purtell commented on HBASE-9857: --- Ah right, sounds good Nick. Blockcache prefetch option -- Key: HBASE-9857 URL: https://issues.apache.org/jira/browse/HBASE-9857 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 9857.patch, 9857.patch, HBASE-9857-0.98.patch, HBASE-9857-trunk.patch, HBASE-9857-trunk.patch Attached patch implements a prefetching function for HFile (v3) blocks, if indicated by a column family or regionserver property. The purpose of this change is to as rapidly after region open as reasonable warm the blockcache with all the data and index blocks of (presumably also in-memory) table data, without counting those block loads as cache misses. Great for fast reads and keeping the cache hit ratio high. Can tune the IO impact versus time until all data blocks are in cache. Works a bit like CompactSplitThread. Makes some effort not to stampede. I have been using this for setting up various experiments and thought I'd polish it up a bit and throw it out there. If the data to be preloaded will not fit in blockcache, or if as a percentage of blockcache it is large, this is not a good idea, will just blow out the cache and trigger a lot of useless GC activity. Might be useful as an expert tuning option though. Or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004793#comment-14004793 ] ramkrishna.s.vasudevan commented on HBASE-11039: Let me assign to me. Will discuss internally and come up with a patch. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Priority: Critical Fix For: 0.99.0, 0.98.3 Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11222) Add integration test for visibility deletes
[ https://issues.apache.org/jira/browse/HBASE-11222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004794#comment-14004794 ] ramkrishna.s.vasudevan commented on HBASE-11222: Sorry for creating a dup. Thanks Andy. Add integration test for visibility deletes --- Key: HBASE-11222 URL: https://issues.apache.org/jira/browse/HBASE-11222 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Once HBASE-10885 gets checked in, create integration tests to verify the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7456) Stargate's HTablePool maxSize is hard-coded at 10, too small for heavy loads
[ https://issues.apache.org/jira/browse/HBASE-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004797#comment-14004797 ] Andrew Purtell commented on HBASE-7456: --- Commit on hold until the SVN-GIT migration is complete. Stargate's HTablePool maxSize is hard-coded at 10, too small for heavy loads Key: HBASE-7456 URL: https://issues.apache.org/jira/browse/HBASE-7456 Project: HBase Issue Type: Bug Components: REST Affects Versions: 0.94.4, 0.95.2 Reporter: Chip Salzenberg Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: HBASE-7456-0.94.patch, HBASE-7456-trunk.patch Please allow the Configuration to override the hard-coded maxSize of 10 for its HTablePool. Under high loads, 10 is too small. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8039) Make HDFS replication number configurable for a column family
[ https://issues.apache.org/jira/browse/HBASE-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8039: -- Fix Version/s: (was: 0.98.3) 0.98.4 Pushing to 0.98.4. Would be a minor effort to implement but do we still have a need? Or we can resolve this as 'Later' or 'Not A Problem'? Make HDFS replication number configurable for a column family - Key: HBASE-8039 URL: https://issues.apache.org/jira/browse/HBASE-8039 Project: HBase Issue Type: Improvement Components: HFile Reporter: Maryann Xue Priority: Minor Fix For: 0.99.0, 0.98.4 To allow users to decide which column family's data is more important and which is less important by specifying a replica number instead of using the default replica number. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[
https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004804#comment-14004804
]
Andrew Purtell commented on HBASE-11165:
bq. HBASE-4246 has related experience.
Thanks for the pointer.
{quote}
then, it called listChildren on the /hbase/unassigned znode, and crashed
with Packet len6080218 is out of range! since the IPC response was larger
than the default maximum.
{quote}
Oh I see that looks like a ZK API limitation then, we can't do partial
enumeration. So this is an opportunity for an alternative [~mantonov]
Scaling so cluster can host 1M regions and beyond (50M regions?)
Key: HBASE-11165
URL: https://issues.apache.org/jira/browse/HBASE-11165
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Attachments: HBASE-11165.zip
This discussion issue comes out of Co-locate Meta And Master HBASE-10569
and comments on the doc posted there.
A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M
regions maybe even 50M later. This issue is about discussing how we will do
that (or if not 50M on a cluster, how otherwise we can attain same end).
More detail to follow.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise
[ https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11039: --- Fix Version/s: (was: 0.98.3) 0.98.4 Moving to 0.98.4 along with the feature it tests. [VisibilityController] Integration test for labeled data set mixing and filtered excise --- Key: HBASE-11039 URL: https://issues.apache.org/jira/browse/HBASE-11039 Project: HBase Issue Type: Test Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.99.0, 0.98.4 Create an integration test for the VisibilityController that: 1. Create several tables of test data 2. Assign a set of auths to each table. Label all entries in the table with appropriate visibility expressions. Insure that some data in every table overlaps with data in other tables at common row/family/qualifier coordinates. Generate data like ITBLL so we can verify all data present later. 3. Mix the data from the different tables into a new common table 4. Verify for each set of auths defined in step #2 that all entries found in the source table can be found in the common table. Like the ITBLL verification step but done N times for each set of auths defined in step #2. 5. Choose one of the source tables. Get its set of auths. Perform a deletion with visibility expression from the common table using those auths. 6. Verify that no data in the common table with the auth set chosen in #5 remains. A simple row count with the set of auths chosen in #5 that should return 0. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10919) [VisibilityController] ScanLabelGenerator using LDAP
[ https://issues.apache.org/jira/browse/HBASE-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10919: --- Fix Version/s: (was: 0.98.3) 0.98.4 Moving to 0.98.4. We don't have time for this to make the .3 release. [VisibilityController] ScanLabelGenerator using LDAP Key: HBASE-10919 URL: https://issues.apache.org/jira/browse/HBASE-10919 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Fix For: 0.99.0, 0.98.4 Attachments: slides-10919.pdf A ScanLabelGenerator that queries an external service, using the LDAP protocol, for a set of attributes corresponding to the principal represented by the request UGI, and converts any returned in the response to additional auths in the effective set. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11124) Pluggable major compaction strategy
[ https://issues.apache.org/jira/browse/HBASE-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11124: --- Fix Version/s: (was: 0.98.3) Pluggable major compaction strategy --- Key: HBASE-11124 URL: https://issues.apache.org/jira/browse/HBASE-11124 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Fix For: 0.99.0 Consider adding to the master a schedulable internal task that manages major compaction. The scheduler implementation should be pluggable. The default plugin should implement the current policy, a simple interval trigger governed by a site configuration property. Reserve a znode as a store for major compaction scheduler state, allow the plugin access to (re)create it and update as needed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11063) Normalize logging between AccessController and VisibilityController
[ https://issues.apache.org/jira/browse/HBASE-11063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11063: --- Fix Version/s: (was: 0.98.3) 0.98.4 Normalize logging between AccessController and VisibilityController --- Key: HBASE-11063 URL: https://issues.apache.org/jira/browse/HBASE-11063 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Fix For: 0.99.0, 0.98.4 The AccessController and VisibilityController should log auditable events using a format parsable with the same regular expression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11149) Wire encryption is broken
[ https://issues.apache.org/jira/browse/HBASE-11149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004817#comment-14004817 ] Andrew Purtell commented on HBASE-11149: Going to commit this when the SVN-GIT migration is finished. Wire encryption is broken - Key: HBASE-11149 URL: https://issues.apache.org/jira/browse/HBASE-11149 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.99.0, 0.98.3 Attachments: 11149-1.txt Upon some testing with the QOP configuration (hbase.rpc.protection), discovered that RPC doesn't work with integrity and privacy values for the configuration key. I was using 0.98.x for testing but I believe the issue is there in trunk as well (haven't checked 0.96 and 0.94). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11057) Improve TestShell coverage of grant and revoke comamnds
[ https://issues.apache.org/jira/browse/HBASE-11057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11057: --- Fix Version/s: (was: 0.98.3) 0.98.4 Improve TestShell coverage of grant and revoke comamnds --- Key: HBASE-11057 URL: https://issues.apache.org/jira/browse/HBASE-11057 Project: HBase Issue Type: Test Reporter: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.4 The TestShell coverage of grant and revoke commands doesn't seem sufficient to catch a botch that prevented global grants. Also cover the alternative grant syntax introduced in HBASE-11001. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11088) Support Visibility Expression Deletes in Shell
[ https://issues.apache.org/jira/browse/HBASE-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11088: --- Fix Version/s: (was: 0.98.3) 0.98.4 Moving to 0.98.4 along with the base feature. Support Visibility Expression Deletes in Shell -- Key: HBASE-11088 URL: https://issues.apache.org/jira/browse/HBASE-11088 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-11058.patch, HBASE-11058_2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10414) Distributed replay should re-encode WAL entries with its own RPC codec
[ https://issues.apache.org/jira/browse/HBASE-10414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10414: --- Fix Version/s: (was: 0.98.3) 0.98.4 Distributed replay should re-encode WAL entries with its own RPC codec -- Key: HBASE-10414 URL: https://issues.apache.org/jira/browse/HBASE-10414 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0, 0.99.0 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.4 HBASE-10412 allows distributed replay to send WAL entries with tags intact between RegionServers by substituting a WALCodec directly for the RPC codec. We should instead have distributed replay handle WAL entries including tags with its own tag-aware RPC codec and drop the direct use of WALCodecs for that purpose. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9527) Review all old api that takes a table name as a byte array and ensure none can pass ns + tablename
[ https://issues.apache.org/jira/browse/HBASE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9527: -- Fix Version/s: (was: 0.98.3) Review all old api that takes a table name as a byte array and ensure none can pass ns + tablename -- Key: HBASE-9527 URL: https://issues.apache.org/jira/browse/HBASE-9527 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Fix For: 0.99.0 Go over all old APIs that take a table name and ensure that it is not possible to pass in a byte array that is a namespace + tablename; instead throw an exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11048) Support setting custom priority per client RPC
[ https://issues.apache.org/jira/browse/HBASE-11048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004819#comment-14004819 ] Andrew Purtell commented on HBASE-11048: Push to 0.98.4? Support setting custom priority per client RPC -- Key: HBASE-11048 URL: https://issues.apache.org/jira/browse/HBASE-11048 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.99.0, 0.98.2 Reporter: Jesse Yates Assignee: Jesse Yates Labels: Phoenix Fix For: 0.99.0, 0.98.3 Attachments: hbase-11048-trunk-v0.patch Servers have the ability to handle custom rpc priority levels, but currently we are only using it to differentiate META/ROOT updates from replication and other 'priority' updates (as specified by annotation tags per RS method). However, some clients need the ability to create custom handlers (e.g. PHOENIX-938) which can really only be cleanly tied together to requests by the request priority. The disconnect is in that there is no way for the client to overwrite the priority per table - the PayloadCarryingRpcController will always just set priority per ROOT/META and otherwise just use the generic priority. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10184: --- Fix Version/s: (was: 0.98.3) [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9804) Startup option for holding user table deployment
[ https://issues.apache.org/jira/browse/HBASE-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9804: -- Fix Version/s: (was: 0.98.3) 0.98.4 Startup option for holding user table deployment Key: HBASE-9804 URL: https://issues.apache.org/jira/browse/HBASE-9804 Project: HBase Issue Type: New Feature Affects Versions: 0.98.0 Reporter: Andrew Purtell Priority: Minor Fix For: 0.99.0, 0.98.4 Introduce a boolean configuration option, false by default, that if set to 'true' will cause the master to set all user tables to disabled state at startup. From there, individual tables can be onlined as normal. Add a new admin method HBA#enableAll for convenience, also a new HBA#disableAll for symmetry. Add shell support for sending those new admin commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10499) In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException
[
https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004827#comment-14004827
]
Andrew Purtell commented on HBASE-10499:
Where are we with this issue?
In write heavy scenario one of the regions does not get flushed causing
RegionTooBusyException
--
Key: HBASE-10499
URL: https://issues.apache.org/jira/browse/HBASE-10499
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-10499.patch,
hbase-root-master-ip-10-157-0-229.zip,
hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log,
master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump,
workloada_0.98.dat
I got this while testing 0.98RC. But am not sure if it is specific to this
version. Doesn't seem so to me.
Also it is something similar to HBASE-5312 and HBASE-5568.
Using 10 threads i do writes to 4 RS using YCSB. The table created has 200
regions. In one of the run with 0.98 server and 0.98 client I faced this
problem like the hlogs became more and the system requested flushes for those
many regions.
One by one everything was flushed except one and that one thing remained
unflushed. The ripple effect of this on the client side
{code}
com.yahoo.ycsb.DBException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
54 actions: RegionTooBusyException: 54 times,
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:245)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
54 actions: RegionTooBusyException: 54 times,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171)
at
org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:232)
... 2 more
{code}
On one of the RS
{code}
2014-02-11 08:45:58,714 INFO [regionserver60020.logRoller] wal.FSHLog: Too
many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s):
97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2,
01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e,
127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253,
29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463,
acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84,
0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e,
d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69,
0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f,
bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b,
cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e,
6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1,
acc43e4b42c1a041078774f4f20a3ff5
..
2014-02-11 08:47:49,580 INFO [regionserver60020.logRoller] wal.FSHLog: Too
many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s):
fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39
{code}
{code}
2014-02-11 09:42:44,237 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
delay of 16689
2014-02-11 09:42:44,237 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
delay of 15868
2014-02-11 09:42:54,238 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
delay of 20847
2014-02-11 09:42:54,238 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
delay of 20099
2014-02-11 09:43:04,238
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10885: --- Fix Version/s: (was: 0.98.3) 0.98.4 Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10499) In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException
[
https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10499:
---
Fix Version/s: (was: 0.98.3)
In write heavy scenario one of the regions does not get flushed causing
RegionTooBusyException
--
Key: HBASE-10499
URL: https://issues.apache.org/jira/browse/HBASE-10499
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-10499.patch,
hbase-root-master-ip-10-157-0-229.zip,
hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log,
master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump,
workloada_0.98.dat
I got this while testing 0.98RC. But am not sure if it is specific to this
version. Doesn't seem so to me.
Also it is something similar to HBASE-5312 and HBASE-5568.
Using 10 threads i do writes to 4 RS using YCSB. The table created has 200
regions. In one of the run with 0.98 server and 0.98 client I faced this
problem like the hlogs became more and the system requested flushes for those
many regions.
One by one everything was flushed except one and that one thing remained
unflushed. The ripple effect of this on the client side
{code}
com.yahoo.ycsb.DBException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
54 actions: RegionTooBusyException: 54 times,
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:245)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
54 actions: RegionTooBusyException: 54 times,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171)
at
org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:232)
... 2 more
{code}
On one of the RS
{code}
2014-02-11 08:45:58,714 INFO [regionserver60020.logRoller] wal.FSHLog: Too
many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s):
97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2,
01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e,
127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253,
29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463,
acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84,
0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e,
d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69,
0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f,
bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b,
cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e,
6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1,
acc43e4b42c1a041078774f4f20a3ff5
..
2014-02-11 08:47:49,580 INFO [regionserver60020.logRoller] wal.FSHLog: Too
many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s):
fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39
{code}
{code}
2014-02-11 09:42:44,237 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
delay of 16689
2014-02-11 09:42:44,237 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
delay of 15868
2014-02-11 09:42:54,238 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
delay of 20847
2014-02-11 09:42:54,238 INFO [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
delay of 20099
2014-02-11 09:43:04,238 INFO [regionserver60020.periodicFlusher]
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004837#comment-14004837 ] Andrew Purtell commented on HBASE-8: So do we include the Java protobuf library into our code under the desired (presume eventually shaded) package, as a temporary measure? Need to resolve the blocker and issue for downstreamers. Should proceed with a patch for that. non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0, 0.98.3 Attachments: shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11114) Backport HBASE-10926 (Use global procedure to flush table memstore cache) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-4: --- Fix Version/s: (was: 0.98.3) 0.98.4 Push to 0.98.4. The compatibility issues will need to be well tested. Backport HBASE-10926 (Use global procedure to flush table memstore cache) to 0.98 - Key: HBASE-4 URL: https://issues.apache.org/jira/browse/HBASE-4 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.4 Backport HBASE-10926 to 0.98. Description from original issue: Currently, user can trigger table flush through hbase shell or HBaseAdmin API. To flush the table cache, each region server hosting the regions is contacted and flushed sequentially, which is less efficient. In HBase snapshot global procedure is used to coordinate and flush the regions in a distributed way. Let's provide a distributed table flush for general use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004830#comment-14004830 ] Andrew Purtell commented on HBASE-10885: Move to 0.98.4. This is about ready but there is shell support and integration test related issues that should go in at the same time. Unlikely all will make the train for .3 but likely all will make it for .4. Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.4 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[ https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004843#comment-14004843 ] Andrew Purtell commented on HBASE-10761: Where are we on this issue? StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic Key: HBASE-10761 URL: https://issues.apache.org/jira/browse/HBASE-10761 Project: HBase Issue Type: Bug Components: Balancer Affects Versions: 0.98.0 Reporter: Victor Xu Fix For: 0.99.0, 0.98.3 Attachments: HBASE_10761.patch, HBASE_10761_v2.patch StochasticLoadBalancer has become the default balancer since 0.98.0. But its balanceCluster method still uses the BaseLoadBalancer.needBalance() which is originally designed for SimpleLoadBalancer. It's all based on the number of regions on the regionservers. This can cause such a problem: when the cluster has some Hot Spot Region, the balance process may not be triggered because the numbers of regions on the RegionServers are averaged. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11194) [AccessController] issue with covering permission check in case of concurrent op on same row
[ https://issues.apache.org/jira/browse/HBASE-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11194: --- Fix Version/s: (was: 0.98.3) 0.98.4 Moving to 0.98.4 as this will require new coprocessor hooks or adjusted hook semantics currently under dev and discussion. [AccessController] issue with covering permission check in case of concurrent op on same row Key: HBASE-11194 URL: https://issues.apache.org/jira/browse/HBASE-11194 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.4 The issue is the hook where we do check in which we have not acquired rowlock. Take case of delete, we do the check in the preDelete() hook. We do get the covering cells and check against their acls. At the point of the preDelete hook, we have not acquired the row lock on the deleting row. Consider 2 parallel threads one doing put and other delete both dealing with same row. Thread 1 acquired the rowlock and decided the TS (HRS time) and doing the memstore write and HLog sync but the mvcc read point is NOT advanced. Thread 2 at same time, doing the delete of the row (Say with latest TS . The intent is to delete entire row) and in place of preDelete hook. There is no row locking happening at this point. As part of covering permission check, it doing a Get. But as said above, the put is not complete and the mvcc advance has not happened. So the Get won’t return the new cell. It will return the old cells. And the check pass for the old cells. Now suppose the new cell ACL is not matching for the deleting user. But the cell was not read, the check has not happened. So the ACL check will allow the user to delete row.. The flow later comes to HRegion#doMiniBatchMutate() and try acquire row lock and by that time the Thread 1 op was over. So it will get lock and will add the delete tombstone. As a result the cell, for which the deleting user has no acl right, also will get deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8039) Make HDFS replication number configurable for a column family
[ https://issues.apache.org/jira/browse/HBASE-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004836#comment-14004836 ] Maryann Xue commented on HBASE-8039: This was meant for use cases that would like to set a smaller number of replications for those less important but more consuming column families. For example, large image files. So I assume how to resolve this issue depends on how we evaluate such use cases. Make HDFS replication number configurable for a column family - Key: HBASE-8039 URL: https://issues.apache.org/jira/browse/HBASE-8039 Project: HBase Issue Type: Improvement Components: HFile Reporter: Maryann Xue Priority: Minor Fix For: 0.99.0, 0.98.4 To allow users to decide which column family's data is more important and which is less important by specifying a replica number instead of using the default replica number. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11212) Fix increment index in KeyValueSortReducer
[
https://issues.apache.org/jira/browse/HBASE-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004845#comment-14004845
]
Andrew Purtell commented on HBASE-11212:
Test failure is unrelated. +1 for commit to trunk and 0.98. Commit on hold
until the SVN-GIT migration is finished.
Fix increment index in KeyValueSortReducer
--
Key: HBASE-11212
URL: https://issues.apache.org/jira/browse/HBASE-11212
Project: HBase
Issue Type: Bug
Reporter: Gustavo Anatoly
Assignee: Gustavo Anatoly
Priority: Minor
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-11212.patch
The index is never incremented inside the loop, therefore context.setStatus
also is never set.
{code}
int index = 0;
for (KeyValue kv: map) {
context.write(row, kv);
if (index 0 index % 100 == 0) context.setStatus(Wrote + index);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-8039) Make HDFS replication number configurable for a column family
[ https://issues.apache.org/jira/browse/HBASE-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8039: -- Fix Version/s: (was: 0.98.4) Thanks [~maryannxue]. I will unschedule this from 0.98 because LOB seems like a trunk thing. Feel free to retarget the fix version if there's a use for this change in isolation. Make HDFS replication number configurable for a column family - Key: HBASE-8039 URL: https://issues.apache.org/jira/browse/HBASE-8039 Project: HBase Issue Type: Improvement Components: HFile Reporter: Maryann Xue Priority: Minor Fix For: 0.99.0 To allow users to decide which column family's data is more important and which is less important by specifying a replica number instead of using the default replica number. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11209) Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4
[ https://issues.apache.org/jira/browse/HBASE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11209: --- Attachment: HBASE-11209.patch A suitable patch. Will commit with credit to N after the SVN-GIT migration is complete. Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4 - Key: HBASE-11209 URL: https://issues.apache.org/jira/browse/HBASE-11209 Project: HBase Issue Type: Brainstorming Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11209.patch On a YCSB test, I saw a 33% performance increase, both on the max latency and on the throughput. I'm convinced enough that this value is better that I think it makes sense to change it on 0.98 as well. More fundamentally, but outside of the scope of this patch, I think this parameter should be changed to something at the region server level: today, we have: - global memstore check: if we're other 40%, we flush the biggest memstore - local: no more than 2 (proposed: 4) memstore size per region. But if we have enough memory and a spike on a region, there is no reason for not taking the write. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11209) Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4
[ https://issues.apache.org/jira/browse/HBASE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004859#comment-14004859 ] Nicolas Liochon commented on HBASE-11209: - It's ok, I can do the commit :-), I'm waiting as well for the migration. If the next 0.98 release is pressing, I have another setting that I think should be changed. Let me create a jira for this right now. Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4 - Key: HBASE-11209 URL: https://issues.apache.org/jira/browse/HBASE-11209 Project: HBase Issue Type: Brainstorming Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11209.patch On a YCSB test, I saw a 33% performance increase, both on the max latency and on the throughput. I'm convinced enough that this value is better that I think it makes sense to change it on 0.98 as well. More fundamentally, but outside of the scope of this patch, I think this parameter should be changed to something at the region server level: today, we have: - global memstore check: if we're other 40%, we flush the biggest memstore - local: no more than 2 (proposed: 4) memstore size per region. But if we have enough memory and a spike on a region, there is no reason for not taking the write. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11219) HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop
[ https://issues.apache.org/jira/browse/HBASE-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004861#comment-14004861 ] Andrew Purtell commented on HBASE-11219: bq. yet I don't know how big win it is here, as seems these buffers should be collected in young gen anyway? You would think but I ran GC manually before dumping the data in the attachment to HBASE-11165. bq. If the approach is good, same technique can be applied to RegionSpecifier.Builder. Looks like an improvement, can we do the same for this other builder in one patch on this issue? Then +1 for trunk and 0.98. HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop --- Key: HBASE-11219 URL: https://issues.apache.org/jira/browse/HBASE-11219 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.99.0, 0.98.3 Attachments: 11219-v1.txt As Andrew showed in the attachment to HBASE-11165, ClusterStatusProtos$RegionLoad$Builder took 25MB heap space. buildServerLoad() calls createRegionLoad() for each region on the region server. One instance of ClusterStatusProtos$RegionLoad$Builder can be created outside the loop and passed to createRegionLoad(). Thanks Andrew for the finding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11219) HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop
[ https://issues.apache.org/jira/browse/HBASE-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11219: --- Fix Version/s: 0.98.3 0.99.0 HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop --- Key: HBASE-11219 URL: https://issues.apache.org/jira/browse/HBASE-11219 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.99.0, 0.98.3 Attachments: 11219-v1.txt As Andrew showed in the attachment to HBASE-11165, ClusterStatusProtos$RegionLoad$Builder took 25MB heap space. buildServerLoad() calls createRegionLoad() for each region on the region server. One instance of ClusterStatusProtos$RegionLoad$Builder can be created outside the loop and passed to createRegionLoad(). Thanks Andrew for the finding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
Nicolas Liochon created HBASE-11226: --- Summary: Document and increase the default value for hbase.hstore.flusher.count Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.98.2, 0.99.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10560) Per cell TTLs
[ https://issues.apache.org/jira/browse/HBASE-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10560: --- Fix Version/s: (was: 0.98.3) 0.98.5 Assignee: Andrew Purtell This is something I'd like to see so assigning to myself to do in the 0.98.5 timeframe. Per cell TTLs - Key: HBASE-10560 URL: https://issues.apache.org/jira/browse/HBASE-10560 Project: HBase Issue Type: New Feature Affects Versions: 0.98.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.5 Now that we have cell tags, we can optionally store TTLs per cell. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004887#comment-14004887 ] Andrew Purtell commented on HBASE-11226: +1 for 0.98. Document and increase the default value for hbase.hstore.flusher.count -- Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11219) HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop
[ https://issues.apache.org/jira/browse/HBASE-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004888#comment-14004888 ] Andrew Purtell commented on HBASE-11219: Patch v2 looks good. Can you quantify the impact? HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop --- Key: HBASE-11219 URL: https://issues.apache.org/jira/browse/HBASE-11219 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.99.0, 0.98.3 Attachments: 11219-v1.txt, 11219-v2.txt As Andrew showed in the attachment to HBASE-11165, ClusterStatusProtos$RegionLoad$Builder took 25MB heap space. buildServerLoad() calls createRegionLoad() for each region on the region server. One instance of ClusterStatusProtos$RegionLoad$Builder can be created outside the loop and passed to createRegionLoad(). Thanks Andrew for the finding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11209) Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4
[ https://issues.apache.org/jira/browse/HBASE-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004877#comment-14004877 ] Andrew Purtell commented on HBASE-11209: Allright N I leave it to you. I'd like to roll the 0.98.3 RC at the end of this week so hopefully we can have end of month releases for 0.98 again. Increase the default value for hbase.hregion.memstore.block.multipler from 2 to 4 - Key: HBASE-11209 URL: https://issues.apache.org/jira/browse/HBASE-11209 Project: HBase Issue Type: Brainstorming Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11209.patch On a YCSB test, I saw a 33% performance increase, both on the max latency and on the throughput. I'm convinced enough that this value is better that I think it makes sense to change it on 0.98 as well. More fundamentally, but outside of the scope of this patch, I think this parameter should be changed to something at the region server level: today, we have: - global memstore check: if we're other 40%, we flush the biggest memstore - local: no more than 2 (proposed: 4) memstore size per region. But if we have enough memory and a spike on a region, there is no reason for not taking the write. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11219) HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop
[
https://issues.apache.org/jira/browse/HBASE-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11219:
---
Attachment: 11219-v2.txt
javadoc warning was not related:
{code}
[WARNING] Javadoc Warnings
[WARNING] javadoc: warning - Multiple sources of package comments found for
package org.apache.hadoop.hbase.io.hfile
{code}
Patch v2 applies same technique to RegionSpecifier.Builder
HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance
when called in a loop
---
Key: HBASE-11219
URL: https://issues.apache.org/jira/browse/HBASE-11219
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 0.98.3
Attachments: 11219-v1.txt, 11219-v2.txt
As Andrew showed in the attachment to HBASE-11165,
ClusterStatusProtos$RegionLoad$Builder took 25MB heap space.
buildServerLoad() calls createRegionLoad() for each region on the region
server.
One instance of ClusterStatusProtos$RegionLoad$Builder can be created outside
the loop and passed to createRegionLoad().
Thanks Andrew for the finding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-11226: Attachment: 11226.v1.patch Document and increase the default value for hbase.hstore.flusher.count -- Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: 11226.v1.patch HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-11226: Status: Patch Available (was: Open) Document and increase the default value for hbase.hstore.flusher.count -- Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.98.2, 0.99.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: 11226.v1.patch HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11219) HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop
[ https://issues.apache.org/jira/browse/HBASE-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004904#comment-14004904 ] Ted Yu commented on HBASE-11219: I am using jmap to check the region servers in cluster accessible to me. Will report back. HRegionServer#createRegionLoad() should reuse RegionLoad.Builder instance when called in a loop --- Key: HBASE-11219 URL: https://issues.apache.org/jira/browse/HBASE-11219 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.99.0, 0.98.3 Attachments: 11219-v1.txt, 11219-v2.txt As Andrew showed in the attachment to HBASE-11165, ClusterStatusProtos$RegionLoad$Builder took 25MB heap space. buildServerLoad() calls createRegionLoad() for each region on the region server. One instance of ClusterStatusProtos$RegionLoad$Builder can be created outside the loop and passed to createRegionLoad(). Thanks Andrew for the finding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004903#comment-14004903 ] Nick Dimiduk commented on HBASE-11226: -- As I understand it, 2 flusher threads will mean more frequent flushes and thus more, smaller files. Should this be accompanied with an increase to hbase.hstore.blockingStoreFiles ? Basically, it moves the blocking pressure from memstore global limit to compaction queue? Document and increase the default value for hbase.hstore.flusher.count -- Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: 11226.v1.patch HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal
[ https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004905#comment-14004905 ] Nick Dimiduk commented on HBASE-8: -- Still no word on my query to the maven group... non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.LiteralByteString -- Key: HBASE-8 URL: https://issues.apache.org/jira/browse/HBASE-8 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: André Kelpe Priority: Blocker Fix For: 0.99.0, 0.98.3 Attachments: shade_attempt.patch I am running into the problem described in https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a newer version within cascading.hbase (https://github.com/cascading/cascading.hbase). One of the features of cascading.hbase is that you can use it from lingual (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. lingual has a notion of providers, which are fat jars that we pull down dynamically at runtime. Those jars give users the ability to talk to any system or format from SQL. They are added to the classpath programmatically before we submit jobs to a hadoop cluster. Since lingual does not know upfront , which providers are going to be used in a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really clunky and breaks the ease of use we had before. No other provider requires this right now. It would be great to have a programmatical way to fix this, when using fat jars. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004908#comment-14004908 ] Anoop Sam John commented on HBASE-10835: I was/am waiting for the SVN- GIT migration Stack.. Thanks.. DBE encode path improvements Key: HBASE-10835 URL: https://issues.apache.org/jira/browse/HBASE-10835 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, HBASE-10835_V3.patch, HBASE-10835_V4.patch, HBASE-10835_V5.patch Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. Encoder again reads kvs one by one from the buffer and encodes and creates a new buffer. There is no need to have this model now. Previously we had option of no encode in disk and encode only in cache. At that time the read buffer from a HFile block was passed to this and encodes. So encode cell by cell can be done now. Making this change will need us to have a NoOp DBE impl which just do the write of a cell as it is with out any encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11161: - Attachment: HBASE-11161.01.patch Updated patch with basic smoketest. Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11161: - Attachment: HBASE-11161.01-0.98.patch [~apurtell] I'd like to get this one into the 0.98 release too. Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11208) Remove the hbase.hstore.blockingStoreFiles setting
[ https://issues.apache.org/jira/browse/HBASE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-11208: Summary: Remove the hbase.hstore.blockingStoreFiles setting (was: Remove the hbase.hstor.blockingStoreFiles setting) Remove the hbase.hstore.blockingStoreFiles setting -- Key: HBASE-11208 URL: https://issues.apache.org/jira/browse/HBASE-11208 Project: HBase Issue Type: Brainstorming Components: Compaction, regionserver Affects Versions: 0.99.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0 It's a little bit of a provocation, but the rational is: - there are some bugs around the delayed flush. For example, if the periodic scheduler has asked for a delayed flush, and that we need to flush, we will have to wait - if the number of WAL files increases, we won't flush immediately if the blockingFile number has been reached. This impacts the MTTR. - We don't write to limit the compaction impact, but they are many cases where we would want to flush anyway, as the writes cannot wait. - this obviously leads to huge write latency peaks. So I'm questioning this setting, it leads to multiple intricate cases, unpredictable write latency, and looks like a workaround for compaction performances. With all the work done on compaction, I think we can get rid of it. A solution in the middle would be to deprecate it and to set it to a large value... Any opinion before I shoot :-) ? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11161: - Fix Version/s: 0.98.3 0.99.0 Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004920#comment-14004920 ] Nick Dimiduk commented on HBASE-11161: -- cc [~jmhsieh], [~rdblue], [~jamestaylor]. Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004922#comment-14004922 ] Nicolas Liochon commented on HBASE-11226: - yes and no. No, because in theory there is no queue, so changing this setting changes nothing, except if one datanode is having issue. Yes, because in practice I've seen this. There is a limit on the number of files. HBASE-11208 is about removing it, but this is too much of a change for a 0.98 default imho... You can see that the queuing is happening in the logs: if the actual flush size is much larger than the configured flush size, it means that you got queued. Document and increase the default value for hbase.hstore.flusher.count -- Key: HBASE-11226 URL: https://issues.apache.org/jira/browse/HBASE-11226 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 0.98.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0, 0.98.3 Attachments: 11226.v1.patch HBASE-6466 add the possibility to have multiple threads for the flusher. The default value is 1, but this should be incremented to 2 reasons: - I've observed that the flush of a region can be delayed because another is in progress. During a write load, this leads to an increased latency because the memstore size increases and then block the client - if, by accident, a flusher hits a slow or bad datanode, all the flush will have to wait until the timeouts expires. With 2 or more flushers and some luck the other regions will be flushed by the second thread. Lastly this setting is important enough to be documented in the standard hbase site imho. I think it's important enough to go in the .98 branch as well There will be an impact: as the flush won't be queued (or less queued) we may have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11161) Provide example of POJO encoding with protobuf
[ https://issues.apache.org/jira/browse/HBASE-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004921#comment-14004921 ] Andrew Purtell commented on HBASE-11161: +1 for trunk and 0.98 Provide example of POJO encoding with protobuf -- Key: HBASE-11161 URL: https://issues.apache.org/jira/browse/HBASE-11161 Project: HBase Issue Type: Sub-task Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11161.01-0.98.patch, HBASE-11161.01.patch, pbtype_example.patch It would be nice to see how to use the DataType API with some out-of-the-box data serialization tools. This ticket is to provide such an example using Protobuf, since we already ship that dependency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
Nick Dimiduk created HBASE-11227: Summary: Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
[ https://issues.apache.org/jira/browse/HBASE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11227: - Attachment: HBASE-11227.00.patch Trivial patch, applied cleanly on both trunk and 0.98. Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11227.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
[ https://issues.apache.org/jira/browse/HBASE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11227: - Status: Patch Available (was: Open) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11227.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11208) Remove the hbase.hstore.blockingStoreFiles setting
[ https://issues.apache.org/jira/browse/HBASE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004936#comment-14004936 ] Nicolas Liochon commented on HBASE-11208: - BTW, by changing this on a YCSB load test, I divided the max latency by 3 (from 45s to 15s). HBASE-11226 adds another division by 3, and this leads us in 5s max latency. YMMV (obviously), but I like what I'm seeing here :-) Remove the hbase.hstore.blockingStoreFiles setting -- Key: HBASE-11208 URL: https://issues.apache.org/jira/browse/HBASE-11208 Project: HBase Issue Type: Brainstorming Components: Compaction, regionserver Affects Versions: 0.99.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0 It's a little bit of a provocation, but the rational is: - there are some bugs around the delayed flush. For example, if the periodic scheduler has asked for a delayed flush, and that we need to flush, we will have to wait - if the number of WAL files increases, we won't flush immediately if the blockingFile number has been reached. This impacts the MTTR. - We don't write to limit the compaction impact, but they are many cases where we would want to flush anyway, as the writes cannot wait. - this obviously leads to huge write latency peaks. So I'm questioning this setting, it leads to multiple intricate cases, unpredictable write latency, and looks like a workaround for compaction performances. With all the work done on compaction, I think we can get rid of it. A solution in the middle would be to deprecate it and to set it to a large value... Any opinion before I shoot :-) ? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11227) Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring
[ https://issues.apache.org/jira/browse/HBASE-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004931#comment-14004931 ] Nick Dimiduk commented on HBASE-11227: -- [~jmhsieh] mind taking a look? Mention 8- and 16-bit fixed-with encodings in OrderedBytes docstring Key: HBASE-11227 URL: https://issues.apache.org/jira/browse/HBASE-11227 Project: HBase Issue Type: Task Components: documentation Affects Versions: 0.95.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Trivial Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11227.00.patch Per title. -- This message was sent by Atlassian JIRA (v6.2#6252)
